from Hacker News

Azure Appears to Be Full

by estensen on 3/26/20, 9:41 AM with 59 comments

• by sz4kerto on 3/26/20, 10:34 AM

  This is a bit like the financial crisis in 2008.

  In 2008, the idea was that if you bundle up a large bunch of mortgages, then the bundle will have low risk because the chances of everything failing at the same time is low. The cloud is designed so that resource usage spikes of individual customers can always be served because one customer is very small compared to the whole infrastructure.

  However, in some cases, these mortgages/resource spikes become highly correlated.

• by estensen on 3/26/20, 10:53 AM

  I think it's very problematic that a major cloud provider is unable to update their status page, even when this has been ongoing for days. All green ticks here: https://status.azure.com/en-us/status

• by imeron on 3/26/20, 10:49 AM

  I heard from an insider that some Azure services had a 10x growth because of the recent changes in our society. It's not like you can prepare for a 10x hit.

  My personal experience for our AWS CI infra that it's struggling more and more recently. Builds are slower on average than a couple of weeks ago. Maybe those VCPUs are not the same VCPUs as yesterday ;D.

• by dmos62 on 3/26/20, 9:56 AM

  There is something satisfying about the bounds of the cloud metaphor being reached (a cloud can't fill the sky).

• by redwood on 3/26/20, 10:30 AM

  AWS and GCP are not full

• by tasubotadas on 3/26/20, 10:34 AM

  That would explain why I have to wait sometimes ~1h until the Azure DevOps queued pipelines build starts.

• by paulcarroty on 3/26/20, 11:26 AM

  Worked on small Azure setup several weeks ago and probably my experience will be useful for other people:

  Pro:

  - you can use shell in browser

  - traffic is cheaper related to AWS

  - fast 1GbE network

  Cons:

  - VM deploy is VERY slow, 2-3 minutes

  - no ipv6 out the box, you need a balancer(!) and 4-5 non-trivial shell commands

  - attaching new storage was extremely painful experience

  It general Azure feels just like middle cloud service.

• by JackPoach on 3/26/20, 10:15 AM

  Yep, people are finally realizing that 'cloud' isn't something magical and limitless. It's just a bunch of servers, connected together, with each having a limit as to how much data in can store and process.

• by Just1689 on 3/26/20, 10:29 AM

  I think this introduces some interesting points to the DR and BCP conversation.

  Is it a safe bet that we can rely on the cloud to have capacity? Normally I wouldn't doubt it but in this sort of situation is becomes more likely they will be put under capacity stress.

  Will the cloud vendors learn and build slack in? I think they're very lean operations and maybe this kind of slack would damage the profitability too much.

  If the cloud vendors can't guarantee capacity ( I suspect this will be the conclusion ) then what does they mean for our DR and BCP planning?

• by rzmnzm on 3/26/20, 11:10 AM

  That's been a recurring issue with the UK South region ever since they introduced it.

  Theregister even reported on it a couple of years ago

  https://www.theregister.co.uk/2017/05/04/microsoft_azure_cap...

• by tyingq on 3/26/20, 10:27 AM

  I'm watching many struggle also with on-prem VPNs, Citrix, WebEx, and so on. Though there do seem to be honest efforts to shore those up and also try more modern tools. I imagine a lot of stodgy companies will have a much better WFH environment after all the dust settles.

• by swebs on 3/26/20, 10:47 AM

  Are there any good guides on switching from Azure to AWS?

• by abafazi on 3/26/20, 10:48 AM

  I bet AWS or GCP will never have a problem like this because they're not stupid enough to operate at only 20% spare capacity at any given time

• by rcarmo on 3/26/20, 11:59 AM

  Disclaimer: I work for Microsoft. I have no particular info on this, but I did read the article _yesterday_ over breakfast, followed links to complaints, etc., and would like to point out two things:

  1. This appears to be a UK-centric thing (and those datacenters don't have the full Azure portfolio, as can be seen here: https://azure.microsoft.com/en-us/global-infrastructure/serv...)

  2. The very last paragraph on the linked article reads: "Note that Azure is a huge service and it would be wrong to give disproportionate weight to a small number of reports. Most of Azure seems to be working fine. That said, capacity in the UK regions was showing signs of stress even before the current crisis, so it is not surprising that issues are occurring now."

  All of this is public info, so maybe people should read up on facts first? :)