from Hacker News

Horrors of Using Azure Kubernetes Service in Production

by pdeva1 on 8/6/18, 6:45 PM with 136 comments

• by summarity on 8/6/18, 8:15 PM

  My $DAYJOB is leading a team which develops applications and gateways (for the 1k+ employee B2B market) that integrate deeply with Azure, Azure AD and anything that comes with it. We do have Microsoft employees (who work on Azure) on our payroll, too.

  I can tell you, as I'm sure anyone in my team can, that Azure is one big alpha-stage amalgation of half-baked services. I would never ever recommend Azure to literally any organization no matter the size. Seeing our customers struggle with it, us struggle with it, and even MS folks struggle with even the most basic tasks gets tiring really fast. We have so many workarounds in our software for inconsistency, unavailability, questionable security and general quirks in Azure that it's not even funny anymore.

  There are some days where random parts of Azure completely fail, like customers not being able to view resources, role assignments or even their directory config.

  An automatic integration test of one of our apps, which makes heavy use of Azure Resource Management APIs, just fails dozens of times a week not because we have a bug, but because state within Azure didn't propagate (RBAC changes, resource properties) within a timeout of more than 15 minutes!

  Two weeks back, the same test managed to reproducibly produce a state within Azure that completely disabled the Azure Portal resource view. All "blades" in Azure just displayed "unable to access data". Only an ultra-specific sequence of UI interactions and API calls could restore Azure (while uncovering a lot of other issues).

  That is the norm, not the exception. In 1.5 years worth of development, there has never been a single week without an Azure issue robbing us of hours of work just debugging their systems and writing workarounds.

  /rant

  On topic though, we've had good experiences with these k8s runtimes:

  - GKE

  - Rancher + DO

  - IBM Cloud k8s (yeah, I know!)

• by QiKe on 8/7/18, 12:29 AM

  (Eng lead for AKS here) While lots of people have had great success with AKS, we're always concerned when someone has a bad time. In this particular case the AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resulting in the kernel oom (out of memory) killer terminating the Docker daemon and kubelet. As part of this investigation we increased the system reservation for both Docker and kubelet to ensure that in the future if a user over schedules their nodes the kernel will only terminate their applications and not the critical system daemons.

• by paxys on 8/7/18, 12:29 AM

  Worth nothing that both Microsoft and Amazon's Kubernetes offerings are very new (literally weeks since GA). While "officially" ready, it is pretty naive to rely on them for production-critical workloads just yet, at least compared to Google Kubernetes Engine which has been running for years.

  If you absolutely need managed Kubernetes, stick to GCP for now.

• by ageitgey on 8/7/18, 2:02 AM

  Here's a fun fact about Azure Kubernetes:

  1. Deploy your Linux service on k8s with redundant nodes

  2. Create a k8s VolumeClaim and mount it on your nodes to give your application some long-lived or shared disk storage (i.e. for processing user-uploaded files).

  3. Wait until the subtle bugs start to appear in your app.

  Because persistent k8s volumes on Azure are provided by Azure disk storage service behind the scenes, lots of weird Windows-isms apply. And this goes beyond stuff like case insensitivity for file names.

  For example, if a user tries to upload a file called "COM1" or "PRN1", it will blow up with a disk write error.

  Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018!

• by nojvek on 8/7/18, 4:20 PM

  I believe this is a cultural problem with Microsoft. Probably similar to other companies but it was very evident at Microsoft. People responsible to allocate resources (The management chain) rarely dogfood the product.

  While the Engineers and PM would complain a lot about quality issues, management wants to prioritize more features. It was a running joke at Microsoft: No one gets promoted for improving existing things, if you want a quick promo, build a new thing.

  So when you see a bazillion half baked things in Azure. That’s because someone got promoted for building each of those half baked things and moving on to the next big thing.

  Going from 0-90% is the same amount of work as 90-99% and the same amount of work as 99.0% - 99.99%. Making things insanely great is hard and requires a lot of dedicated focus and a commitment to set a higher bar for yourself.

• by hb3b on 8/7/18, 3:08 PM

  I joined a healthcare startup in 2014 that had a small infrastructure on Azure. Back then AWS weren't signing BAAs and Azure was the only player in town. Being an early startup, the company didn't purchase a support plan from Azure. One day Azure suffered a major outage (may have been storage related) and over an hour later, I reached out to Microsoft for written confirmation that we could forward to customers. Since we didn't have a support plan they flat-out refused to provide any documentation whatsoever about the issue. They wanted $10,000.

  Azure - never again. Company moved to AWS within a quarter.

• by mgalgs on 8/6/18, 11:23 PM

  FWIW, Amazon's hosted Kubernetes offering (EKS) isn't stable either (DNS failures, HPA is known to be broken, etc.).

• by curiousDog on 8/6/18, 11:01 PM

  This is sad. From what I hear, one of the founders of k8s works on AKS.

  Only a matter of time before GCP becomes the #1/2 cloud provider.

• by taherchhabra on 8/7/18, 2:03 AM

  Had a similar experience with azure cosmos graph API. The API is half baked. Doesn't support all gremlin operations. Even supported operations give non standard output. Switched to aws Neptune immediately when they launched

• by AaronFriel on 8/7/18, 2:18 AM

  There are definitely growing pains with using Kubernetes on Azure. I've wondered a few times if other platforms have similar issues and have seen more than a few complaints about EKS.

  Microsoft has some great people working on Azure, but I do feel like AKS was released to GA too soon. Without a published roadmap and scant acknowledgment of issues, I'm not sure I could recommend it to my clients or employer. It's disappointing, because I've had few issues with other Azure services.

  Full disclosure: I receive a monthly credit through a Microsoft program for Azure.

• by rcconf on 8/6/18, 7:48 PM

  Hmm, this isn't great. Currently using Azure Kubernetes Service and we haven't had many issues so far, but we just made the shift.

  Hope I don't have to move over to Google cloud.

• by parasubvert on 8/9/18, 3:42 PM

  DNS failures were almost certainly related to all the k8s system services on the cluster not having CPU or memory reservations, and KubeDNS was flaking.

  In general AKS is a vanilla k8s cluster and expects you know what you’re doing. MS arguably should enforce some opinions about how things like system services have reservations, etc, but none of this is vanilla. The trouble is that K8s defaults are pretty poor from a security (no seccomp profiles or apparmor/se profiles) and performance perspective (no reservations on key system DaemonSets).

  We’ve had this interesting industry pendulum swing between extreme poles of “we hate opinionated platforms! Give me all the knobs!” And “this is too hard, we need opinions and guard rails!”. I think the success of K8s is exposing people to the complexity of supplying all of the config details yourself and we will see a new breed of opinionated platforms on top of it very shortly. It reminds me of the early Linux Slackware and SLS and Debian days where people traded X11 configs and window manager configs like they were treasured artifacts before Red Hat, Gnome and KDE, SuSE, and eventually Ubuntu, started to force opinions.

• by spicyusername on 8/6/18, 11:09 PM

  This is probably why they're releasing OpenShift on Azure. To let Red Hat engineers manage the kubernetes part.

  https://azure.microsoft.com/en-us/blog/openshift-on-azure-th...

• by FlorianRappl on 8/7/18, 6:32 PM

  We have a larger migration project going on for months. So far not a single failure occurred and our TEST environment is already fully migrated (quite responsive and rock solid) since 2 weeks.

  However, I do share that Azure indeed has released a lot of half-baked features and services lately (last 1.5 to 2 years). I hope this trend does not continue.

• by stefanatfrg on 8/7/18, 8:41 AM

  Couple of questions to the OP:

  1. What version of docker / container runtime is being used?

  2. What base image for your containers is being used? eg. alpine has known DNS issues [1]

  [1] https://www.youtube.com/watch?v=ZnW3k6m5AY8

• by bsaul on 8/7/18, 9:52 AM

  Side question : what are the best practices for development ? Are you suppose to run a local kubernetes deployment ( it looks like it's pretty hard to set up) , or do you run everything outside of containers when developping and then deal with k8 packaging and deployment as a completely separated issue ( which looks like it could lead to discovering a lot of issues on the preproduction environment) ?

• by gercheq on 8/7/18, 8:28 AM

  Azure is not bad but there are definitely some rough edges. We're having trouble with their Bizspark Sponsorship biling https://news.ycombinator.com/item?id=17698948

• by rdl on 8/7/18, 4:34 AM

  Key Vault (their HSM product) is even worse.

• by ubuntunero on 8/7/18, 2:30 AM

  interesting. thanks

• by partiallypro on 8/7/18, 2:14 AM

  It's a very new offering, the Linux App Services are still in beta, I have no idea why you would roll it into production expecting no hiccups. AWS is also new on this. Give it 6 months and let the kinks work out before migrating workloads. Seems like common sense.