by picozeta on 6/1/22, 12:19 PM with 416 comments
1) internal users: mainly developers by providing CI/CD
2) external users: end users
The platform of choice is mostly Kubernetes these days, which promises among other things stuff like
- load balancing
- self-healing
- rollbacks/rollouts
- config management
In particular, a main critic is "state" and the possibility to change that state by e.g. messing with config files via SSH, which makes running and maintaining these servers more error-prone.
However, my main question is:
"If this old way of doing things is so error-prone, and it's easier to use declarative solutions like Kubernetes, why does the solution seem to need sooo much work that the role of DevOps seems to dominate IT related job boards? Shouldn't Kubernetes reduce the workload and need less men power?"
Don't get me wrong, the old way does indeed look messy, I am just wondering why there is a need for so much dev ops nowadays ...
Thanks for your answers.
by jmillikin on 6/1/22, 12:39 PM
> 1) internal users: mainly developers by providing CI/CD
> 2) external users: end users
>
> Nowadays we call people that do 1) DevOps and people that do
> 2) SREs (so one could argue that the role of sys admins just
> got more specialized).
SRE is a specialized software engineering role -- you'd hire SREs if you wanted to create something like Kubernetes in-house, or do extensive customization of an existing solution. If you hire an SRE to do sysadmin work, they'll be bored and you'll be drastically overpaying.
DevOps is the idea that there shouldn't be separate "dev" and "ops" organizations, but instead that operational load of running in-house software should be borne primarily by the developers of that software. DevOps can be considered in the same category as Scrum or Agile, a way of organizing the distribution and prioritization of tasks between members of an engineering org.
---
With this in mind, the question could be reframed as: if projects such as Kubernetes are changing the nature of sysadmin work, why has that caused more sysadmin jobs to exist?
I think a general answer is that it's reduced the cost associated with running distributed software, so there are more niches where hiring someone to babysit a few hundred VMs is profitable compared to a team of mainframe operators.
by binarymax on 6/1/22, 12:58 PM
If you need less than 8 instances to do host your product, run far away anytime anyone mentions k8s
by mancerayder on 6/1/22, 2:58 PM
Yes, it's quite complicated.
No, an API to control a managed EKS/GCK cluster + terraform + Jenkins/Azure DevOps/etc. does not mean that magically the developer can 'just deploy' and infrastructure jobs are obsoleted. That's old AWS marketing nonsense predating Kubernetes.
There's a whole maintenance of the CI/CD factory and its ever demanding new requirements around performance, around Infosec requirements, around scale, and around whatever unique business requirements throw a wrench in the operation.
Sticking to ECS I guess is a valid point. What Kubernetes gives you is a more sophisticated highly available environment built for integration (Helm charts and operators and setups that when they work give you more levers to control resources allocations, separations of app environments, etc.)
And as an aside, I've been doing this for 20 years and long before Kubernetes, before Docker, hell, before VMs were used widely in production, I observed the developer mindset: Oh but it's so easy, just do X. Here, let me do it. Fast forward a year of complexity later, you start hiring staff to manage the mess, the insane tech debt the developers made unwittingly, and you realize managing infrastructure is an art and a full time job.
A story that is visible with many startups that suddenly need to make their first DevOps hire, who in turn inherit a vast amount of tech debt and security nightmares.
Get out of here with, it's just API calls. DevOps jobs aren't going away. It's just the DevOps folks doing those API calls now.
by mynameisash on 6/1/22, 11:15 PM
Holy smokes, did that thing blow up. A pod would go down, get stuck in some weird state (I don't recall what anymore), and K8s would spin a new one up. Okay, so it was running, but with ever-increasing zombie pods. Whatever. Then one pod would get in such a bad state that I had to nuke all pods. Fortunately, K8s was always able to re-create them once I deleted them. But I was literally deleting all my pods maybe six or seven times per day in order to keep the service up.
Ultimately, I rewrote the whole thing with a simplified architecture, and I vowed to keep clear of K8s for as long as possible. What a mess.
by jeffwask on 6/1/22, 1:39 PM
In the olden days of 10 years ago, most operations teams worked around the clock to service the application. Like every day there would be someone on my team doing something after hours usually multiple. Tools like Kubernettes, Cloud (AWS, GCP, Azure) have added significant complexity but moved operations to more of a 9 to 5 gig. Less and less do I see after hours deployments, weekend migrations, etc. Even alert fatigue goes way down because things are self healing. This is on top of being able to move faster and safer, scale instantly, and everything else.
Operations side used to be a lot of generalists admin types and DBA's. With today's environment, you need a lot more experts. AWS alone has 1 trillion services and 2.4 billion of those are just different ways to deploy containers. So you see a lot more back end roles because it's no longer automate spinning up a couple servers, install some software, deploy, monitor and update. It's a myriad of complex services working together in an ephemeral environment that no one person understands anymore.
by etruong42 on 6/1/22, 12:52 PM
"since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added." Planet Money, May 17, 2017, Episode 606: Spreadsheets!
by devonkim on 6/1/22, 8:26 PM
A lot of this stuff really is trying to address the core problem we've had for a long time that probably won't ever end - "works fine on my computer."
by majewsky on 6/1/22, 12:38 PM
That's valuable because, on the scale of large companies, it's much easier to hire "a network expert" or "a storage expert" or even "a Gatekeeper policy writing expert" than to hire a jack of all trades that can do all of these things reasonably well.
The corollary from this observation is that Kubernetes makes much less sense when you're operating at a start-up scale where you need jacks of all trades anyway. If you have a team of, say, 5 people doing everything from OS level to database to web application at once, you won't gain much from the abstractions that Kubernetes introduces, and the little that you gain will probably be outweighed by the cost of the complexities that lurk behind these abstractions.
by moshloop on 6/1/22, 2:02 PM
The goal of Kubernetes is to improve the portability of people by introducing abstraction layers at the infrastructure layer - These abstractions can seem overly complex, but they are essential to meet the needs of all users (developers, operators, cloud providers, etc)
Before kubernetes in order for a developer to deploy an application they would need to (send email, create terraform/cloudformation, run some commands, create ticket for loadbalancer team, etc) - these steps would rarely be same between companies or even between different teams in the same company.
After kubernetes you write a Deployment spec, and knowing how to write a deployment spec is portable to the next job. Sure there are many tools that introduce opinionated workflows over the essentially verbose configuration of base Kubernetes objects, and yes your next job may not use them, but understanding the building blocks, still make it faster than if every new company / team did everything completely differently.
If you only have a single team/application with limited employee churn - then the benefits may not outweigh the increased complexity.
by habitue on 6/1/22, 6:03 PM
1. What people expect: less work needs to be done to get what you had before.
2. What people don't expect: more is expected because what used to be hard is now simple
So while it may have taken a few weeks to set up a pet server before and as a stretch goal you may have made your app resilient to failures with backoff retry loops etc. Now that's a trivial feature of the infrastructure, and you get monitoring with a quick helm deploy. The problems haven't disappeared, you're just operating on a different level of problems now. Now you have to worry about cascading failures, optimizing autoscaling to save money. You are optimizing your node groups to ensure your workloads have enough slack per machine to handle bursts of activity, but not so much slack that most of your capacity is wasted idling.
Meanwhile, your developers are building applications that are more complex because the capabilities are greater. They have worker queues that are designed to run on cheap spot instances. Your CI pipelines now do automatic rollouts, whereas before you used to hold back releases for 3 months because deploying was such a pain.
Fundamentally, what happens when your tools get better is you realize how badly things were being done before and your ambition increases.
by rconti on 6/1/22, 11:58 PM
It's like asking "if the computer saves us all so much work, why do we have more people building computers than we ever had building typewriters"?
Something can "save labor" and still consume more labor in aggregate due to growth.
by jljljl on 6/1/22, 4:38 PM
> Kubernetes is a platform for building platforms. It's a better place to start; not the endgame.
https://twitter.com/kelseyhightower/status/93525292372179353...
Everywhere I've worked, having developers use and develop Kubernetes directly has been really challenging -- there's a lot of extra concepts, config files, and infrastructure you have to manage to do something basic, so Infra teams spend a lot of resources developing frameworks to reduce developer workloads.
The benefits of Kubernetes for scalability and fault tolerance are definitely worth the cost for growing companies, but it requires a lot of effort, and it's easy to get wrong.
Shameless plug: I recently cofounded https://www.jetpack.io/ to try and build a better platform on Kubernetes. If you're interested in trying it out, you can sign up on our website or email us at `demo [at] jetpack.io`.
by zelphirkalt on 6/1/22, 12:58 PM
The longer answer is: When you switch to Kubernetes, you are introducing _a lot_ of complexity, which, depending on your actual project, might not be inherent complexity. Yes, you get a shiny tool, but you also get a lot of more things to think about and to manage, to run that cluster, which in turn will require, that you get more devops on board.
Sure, there might be projects out there, where Kubernetes is the right solution, but before you switch to it, have a real long hard thinking about that and definitely explore simpler alternatives. It is not like Kubernetes is the only game in town. It is also not like Google invents any wheels with Kubernetes.
Not everyone is Google or Facebook or whatever. We need to stop adopting solutions just because they get hyped and used at big company. We need to look more at our real needs and avoid introducing unnecessary complexity.
by caymanjim on 6/1/22, 12:48 PM
by bombcar on 6/1/22, 12:31 PM
The number of single-server setups with kubernetes thrown in for added complexity and buzzwords I’ve found is way too dang high.
by MrBuddyCasino on 6/1/22, 12:36 PM
Because we're living in the stone age of DevOps. Feedback cycles take ages, languages are not typed and error prone, pipelines cannot be tested locally, and the field is evolving rapidly like FE javascript did for many years. Also I have a suspicion that the mindset of the average DevOps person has some resistance to actually using code, instead of yaml monstrosities.
There is light at the tunnel though:
- Pulumi (Terraform but with Code)
- dagger.io (modern CI/CD pipelines)
Or maybe the future is something like ReplIt, where you don't have to care about any of that stuff (AWS Lambdas suck btw).
by rahen on 6/1/22, 1:21 PM
All you need is to rewrite your application (think microservices), reduce cold latency (get rid of anything VM based such as Java, or rewrite in Spring or Quarkus), use asynchronous RPC, and decouple compute and storage.
Then you need an elastic platform, for instance Kubernetes, with all the glue around such as Istio, and Prometheus, and Fluentd, and Grafana, Jaeger, Harbor, Jenkins, maybe Vault and Spinnaker.
Then you can finally have your production finely elastic, which 90% of companies do not need. Microservices are less performant, costlier, and harder to develop than n-tiers applications and monoliths, and way harder to debug. They're just better at handling surges and fast scaling.
If what you want is:
- automated, predictable deployments
- stateless, declarative workloads
- something easy to scale
Then Docker Compose and Terraform is all you need.
If you also need orchestration and containers are your goal, then first try Docker Swarm. If you need to orchestrate various loads and containers are a mean and not a goal, then try Nomad.
Finally, if you will need most resources Kubernetes has to offer (kubectl api-resources), then yes, opt for it. Few companies actually have a need for the whole package, yet they have to support its full operational cost.
Most companies just pile up layers, then add yet a few more (Java VMs on top of containers on top of an orchestrator on top of x86 VMs on top of(...)), and barely notice the miserable efficiency of the whole stack. Well it's using Kubernetes, it's now "modernized".
by tapoxi on 6/1/22, 12:48 PM
But DevOps means many things because it's not clearly defined, which also makes it difficult to hire for. It's a "jack-of-all-trades" role that people somehow fell into and decided to do instead of more traditional software engineering.
Also, from what I've experienced from our internship program, CS programs are really bad at covering these fundamentals. Students aren't learning such basics as version control, ci/cd, cloud platforms, linux, etc.
by oxplot on 6/1/22, 2:18 PM
Kubernetes definitely achieves this goal well, and in a relatively portable way. But just like any other engineering decision, you should evaluate the trade offs of learning a completely new OS just to get a simple web site up, versus running a nginx instance with bunch of cgi scripts.
by jedberg on 6/1/22, 10:04 PM
In a small organization, you can get away with a sysadmin running a Kubernetes cluster to enable that. In a larger org you'll need SREs as well as Operations Engineers to build and maintain the tools you need to enable the engineers.
by whalesalad on 6/1/22, 1:24 PM
by codegeek on 6/1/22, 2:10 PM
- Kubernetes is very complex to setup
- It is not needed for many use cases
- It is (hopefully) not the defacto and standard for devops
- Load Balancing is already a solved problem way before Kubernetes. For many use cases, you don't need the complexity. Even things like Self Healing are kinda solved by AWS Auto Scaling for example.
- NOt every use case needs Kubernetes and its additional overhead/complexity
- Most importantly, devops is not "one size fits all" magic wand that Kubernetes or any other tool can solve. Various nuances to consider and hence you need DevOps as a role.
by techthumb on 6/1/22, 4:37 PM
It helps standardize:
- deployments of containers
- health checks
- cron jobs
- load balancing
Is it same/similar across teams within and outside your organization.
If not, what would it cost to build consensus and shared understanding?
How would you build this consensus outside your organization?
For small organizations, one should do whatever makes them productive.
However, as soon as you need to standardize across teams and projects, you can either build your own standards and tooling or use something like K8S.
Once you have K8S, the extensibility feature kicks in to address issues such as:
- Encrypted comms between pods
- Rotating short lived certificates
However, if not K8S then, what alternative should we consider to build consensus and a shared understanding?
by Seriomino on 6/1/22, 3:03 PM
Our current platform is much more stable, has more features, and bigger than what the prev team did.
There are plenty of things you can't see like security or backup or scalability.
Backup were done app by app basis. Now you can do snapshots in k8s.
Security still is a mess. But now you can at least isolate stuff.
Scalability meant installing your application x times manually and configuring load balancer etc. Now you set it up per cluster.
Additional features you get with k8s: Auto scaling, high availability, health checks, self healing, standardization.
A lot of things got invented which lead to k8s like container or yaml.
Now with the operator pattern you can also replace admin and embed operational knowledge I to code.
Infrastructure was not ready to be controlled by code like this ever before.
by viraptor on 6/1/22, 12:34 PM
Kubernetes helps with: networking setup, consistent deployments, task distribution. That's about it. It's more standardised than plain VMs, but you still have to deal with the other 90% of work.
by togaen on 6/1/22, 12:55 PM
by edanm on 6/1/22, 4:09 PM
If k8s is as amazing and time saving as you would imagine, you'd expect many companies to want to adopt it, so you'd expect there to be lots of job postings!
It's like saying "if computers are such time savers, why do so many companies hire people that have knowledge in computers". It's because this is a good tool that companies want to hire people with knowledge in that tool!
by samsk on 6/1/22, 12:51 PM
Now because of new possibilities, and new development they want to switch to Kubernetes to have that new possibilities everyone is talking about, and now you have to build many new containers, configure k8s, autoscalling etc... and developers don't know it (yet) and don't have time to learn it.
So lets hire a DevOps (me) that will do it ;-)
by wvh on 6/1/22, 2:11 PM
A lot of what is called DevOps goes into adapting software to this new mindset. A lot of that software is not written with best practices in mind, and likewise lots of tools are still in their infancy and have rough edges. I think it's fair to say some time and resources go into learning new ways of doing things, and it might not be the best choice for everybody at this stage to spend those resources unless there's an obvious need.
by jstream67 on 6/1/22, 1:15 PM
Kubernetes and 'DEVOPS' are the new hotness at non-FANG companies as they are always 5-10 years behind the trends. Expect to see more of it before it goes out of fashion.
Also DevOps is just a title. Nobody read the book, nobody is trying to create what the original guy at Google or whatever had in mind. It is just a all encompassing job doing the same activities that the sysadmin used to do. HR tells companies that they should rename their sysadmin departments to DevOps departments and everything else continues as normal.
by JohnHaugeland on 6/1/22, 4:07 PM
1. Kubernetes isn't the solution 2. Kubernetes is expensive and extremely maintenance prone 3. Most of the companies I've seen switch to Kube I've seen switch away afterwards
Every time I've seen someone bring up Kubernetes as a solution, everyone at the table with first hand experience has immediately said no, loudly
Remember, there was a time at which someone wouldn't have been laughed out of the room for suggesting Meteor stack, and right now people are taking GraphQL seriously
Kube doesn't make sense until you have hundreds of servers, and devops makes sense at server #2
by bashinator on 6/1/22, 4:41 PM
TL; DR - K8s and cloud let me do the work of ten of my old selves.
by durnygbur on 6/1/22, 1:41 PM
by rbranson on 6/1/22, 2:09 PM
by pythops on 6/1/22, 11:59 PM
by mfer on 6/1/22, 12:44 PM
1) Kubernetes is an infra platform for the ops in DevOps. If developers need to spend a lot of time doing Kubernetes it takes away from their ability/time to do their dev. So, there are a lot of platform teams who pull together tools to simplify the experience for devs or DevOps specialists who handle operating the workloads.
2) Kuberentes is, as Kelsey Hightower puts it, a platform to build platforms. You need DevOps/SREs to do that.
3) Kubernetes is hard. The API is huge and it's complex. The docs are limited.
by raffraffraff on 6/2/22, 6:55 AM
Many years ago I worked at a large, old tech company where all of these areas had dedicated teams.
PS: how many people at your company "really" know Kubernetes inside out? And if it misbehaves, who do you expect to have the answer?
by phoehne on 6/1/22, 3:17 PM
by hsn915 on 6/1/22, 1:14 PM
When technology makes previous difficult things easy, this technology will be used to do more things that were previously impossible.
I personally haven't seen anyone use k8s to achieve things that were impossible before. They just use it because 1) They think everyone is using it 2) They don't know how to do it any other way
by thedougd on 6/1/22, 1:15 PM
by throwaway892238 on 6/1/22, 3:48 PM
It's true that moving away from mutable state is a sea change that is (very) slowly engulfing the industry. But the tide is still very far out. The cloud is a pile of mutable state, leading to the same system instability and maintenance headaches that existed before the cloud, requiring new skills and workers to deal with. Redesigning the cloud to be immutable will take decades of re-engineering by vendors, and even when it's done, we'll still need economies of scale to reduce headcount.
by GuB-42 on 6/1/22, 4:16 PM
Originally I thought it was a methodology assisted by a set of tools to make it easier for devs and ops (sys admins) to work together, mostly by giving both sides the same environment, usually in the form of docker containers.
Admins configure servers, or server pools since physical machines tend to be abstracted these days, set up applications, including the CI/CD stuff, make sure everything is secure, up to date and in working order, etc... Devs write the code, and test it on the platform supplied by the admins.
And now I see all these DevOps jobs. I mean what does it means? You are dev or you are ops, so what does DevOps means? It doesn't mean both, DevOps usually don't write software, so DevOps is ops, maybe we could call them container administrators, like we have database administrators.
I think that the confusion between the DevOps methodology and the DevOps job title give the wrong idea. Someone needs to make all these servers work, calling it serverless just means there is another abstraction layer, and while abstraction has some benefits, it rarely lessens the workload, but it may change job titles, here sysadmin -> DevOps.
by muskmusk on 6/1/22, 4:47 PM
Wages does not count as cost of goods sold, so who cares about a couple of extra hires? Funding is easy.
Also you severely underestimate the amount of work that goes under DevOps. Everything from build servers to test and security infrastructure is usually handled by DevOps. It's a massive surface area and it would be way worse without kubernetes.
by autarchprinceps on 6/1/22, 3:01 PM
by drakonka on 6/1/22, 8:43 PM
by oofnik on 6/2/22, 5:27 AM
Division of labor and specialization are two natural results of any rapidly evolving industry. The tech industry is no exception. The problem is that there are currently comparatively quite a lot of ways to learn the skills necessary to become a competent web developer, backend engineer, data scientist, etc. Compared to these titles, the ways to learn the skills involved in designing, operating, and maintaining scalable cloud infrastructure have not kept up with market demand.
Kubernetes is not "the" solution, but it is one of several solutions to the problem of standardizing a trade skill for the purposes of making it transferrable. Nobody wants to go to work at a place where the skills they need to do their job well are both difficult to acquire and completely useless once they get a new job.
by vegai_ on 6/1/22, 1:08 PM
Yeah, you lost me already. This is a bit like asking why there are other languages besides Java.
by moomin on 6/1/22, 3:26 PM
Now, I’m not saying k8s is as transformative as the word processor, but if it was, you’d probably expect to see more ops people, not fewer. They’d just be doing different things.
by ac50hz on 6/1/22, 1:16 PM
by bg24 on 6/11/22, 7:57 PM
As part of my job, I see thousands of small and medium businesses happily embracing kubernetes. They do not have separate devops or security or infra folks. A few engineers do it all. Yes, there are challenges with so many moving parts and rapid releases with breaking APIs. But expect that to be fixed in coming years.
Why so many DevOps jobs? Are you noticing the jobs being eliminated, or are you just seeing more DevOps jobs being created?
by kodah on 6/2/22, 1:40 AM
A lot of companies moved to the cloud because their old data centers (or hosts) were driven by ticket systems instead of API's and access management. What took three weeks was now solved in seconds; the beginning was fascinating without a doubt. Then companies realized they owned none of this virtualized infrastructure and were at the behest of a very large corporation who could make sweeping changes with little to no notice. Although Kubernetes was pitched as providing extra grease to the gears of the enterprise, and they weren't wrong, that is not its total value to the enterprise.
The real value in Kubernetes running your own platform on someone elses hardware, especially to the degree where you can eventually free yourself from cloud provider lock-in that the above incurred. An example, if a company can spin up a team to create a database-as-a-service in Kubernetes clusters, then your RDS costs can shift dramatically down, and it develops a new level of capability and understanding that your company never had before.
I'm a SRE-SE, but I mostly use the title "Distributed Systems Software Engineer" because I feel that really fits what I do. DevOps is just a catch-all title for non-application-software tasks and roles at this point because it consumed so many things like "release manager", "QA", "application operations", etc... Personally, I do not trust companies or teams that use this as some sort of distinguishable title.
To answer the last part of your question, "Why are DevOps everywhere", because companies have diverse needs in terms of supporting software and software development, and DevOps is basically the catch all email of software engineering.
by hotpotamus on 6/1/22, 12:43 PM
by chazu on 6/1/22, 12:54 PM
Read the introduction to the SRE book, available free online [1] - and you'll see that SRE is defined _in contrast to_ systems administration. Its specifically defined as software engineering with the goal of managing operational complexity.
Modern shops' failure to understand this (most SREs haven't read any of the book, let alone stopped to think what SRE actually means) is IMHO a primary factor in the failure of most "devops transformations"
by aaccount on 6/2/22, 4:13 AM
For example, you don't need a docker container to deploy Mysql, although you can deploy Mysql inside one. But most development processors are so badly managed that one product has many conflicting libraries and dependencies. Eventual requiring each component to be isolated within its own container. Finally leading to an unmanageable number of containers requiring Kubernetes to manage the mess.
by iknownothow on 6/1/22, 12:47 PM
At the end of transitive reduction, you end up with a graph with all redundant dependencies removed but functionally, it is still the same graph.
by kevwil on 6/1/22, 11:01 PM
by raygelogic on 6/1/22, 2:35 PM
by Art9681 on 6/1/22, 5:19 PM
Also, there is more demand than supply. Everyone wants to do Kubernetes and DevOps pipelines but the amount of folks experienced in those fields is small compared with demand.
It requires knowledge in many domains because it abstracts the entire data center. So you can’t just take a mid level sysadmin or developer and expect them to jump right in.
by throwaway894345 on 6/1/22, 2:04 PM
by ltbarcly3 on 6/1/22, 1:26 PM
by jasonshaev on 6/1/22, 3:41 PM
I have no data to back this up, but my hypothesis is that if you zoom out, and look across the entire industry, the % is vanishingly small. It may seem like every company is running or adopting kubernetes within our bubbles but our perspective is biased.
(Note: I'm not espousing an opinion on kubernetes itself, just about it's total adoption across the entire industry and how that effects the number of devops/sysadmin/SRE roles.
by Garlef on 6/1/22, 1:15 PM
-------
Take the development of programming as an analogy:
* Punched cards
* Programming in assembler
* Goto
* Callable procedures
* Proper functions
* Compiled languages (There used to be companies just selling a big C compiler)
* Interpreters/JIT compilation/...
* ...
-------
And here's a similar progression:
* Servers in your basement
* Some server rented where you login via SSH
* Docker/Kubernetes/Clusters in the cloud
* Lambdas and other serverless solutions
* ...
As a sibling comment pointed out: We're still in the stone ages. Somewhere between punch cards and proper functions.
-------
To rephrase it in reversal: "Infrastructure as code has it backwards"
Right now, we manually partition our code, then provision infrastructure and then push our code to this infrastructure.
Instead, we should take a high level language that compiles to the cloud:
Just write your business logic and the the compiler figures out what clusters/services/event-buses/databases/etc to use; It will automatically partition the code, package, build, provision, push, update. And there's even room for something like JIT: Based on the load parts of your logic get, the compiler could switch databases. Also: Automated data migrations based on your code updates. But I guess we'll end up with a big distributed virtual machine that scales infinitely and completely hides the existence of servers.
There's already some glimpses of this future: No-code, the magic pulumi does with lambdas, several language projects that get rid of the file system and just store the AST in a DB, smart contracts where you pay for single computation steps...
-------
But back to the question: Kubernetes/AWS/etc is a lot of work because it's not really THE SOLUTION.
by rufius on 6/1/22, 6:16 PM
However, as with most large platforms, they require ceremonies and priests (devops engineers). Someone has to make the offerings.
Much as people would like to believe, you don’t reduce complexity, you just shuffle it around and there’s an exchange rate. Even with solutions like Fly.io, you’re not getting rid of complexity in aggregate, you’re paying them to manage it (I.e. the exchange rate).
by taylodl on 6/1/22, 1:43 PM
by dtech on 6/1/22, 1:03 PM
I'd wager providing all those things 20 years ago without k8s and CI tools would've required relatively more sysadmins
by mkl95 on 6/1/22, 3:44 PM
by atmosx on 6/2/22, 7:52 PM
Kubernetes buys the org some things but it is complex and you have to know how to write the app in a certain _way_ in order of the app to be "scalable, etc.".
There are no free meals or as someone smarter than me said long time ago "there is no royal road to learning".
by pantulis on 6/1/22, 1:18 PM
The question is: do _you_ need this added complexity? That humble VPS can scale _a lot_ too.
by fertrevino on 6/1/22, 9:50 PM
by wecloudpro on 6/1/22, 9:10 PM
Then, for designing, implementing and maintaining all these extra elements is why you need a devops guy. Also not mentioning how extremely fast things are moving in the cloud era.
by oznog on 6/1/22, 6:47 PM
1. Dethroning sysadmins introducing devops in the middle ("devs" capable of deploying in the cloud but unable to control the OS).
2. Increase CPU and other resource consumption (promoting heavy frameworks, unable to pass the Doherty threshold in 2022).
For clouds, increasing complexity and costs almost always expands the business.
by benlivengood on 6/1/22, 4:16 PM
What I'm trying to do at work is simplify the build ecosystem for all languages to the familiar `configure ; make ; make test ; make install` sequence that works well for OSS. If every ecosystem fit into that metaphor then the loop could be closed pretty effectively by any cloud provider by letting users add repositories to the CI/CD framework and it would do e.g. a standard docker build (configure, make, test), docker push (make install 1/2), and kubectl rollout to k8s at the end (remainder of make install).
Blockers:
Liveness and readiness checks are not automatic; they need to be part of each language*framework so that developers don't have to implement them by hand. At Google they just sort of came with the default HTTPServer class in your language of choice, with callbacks or promises you knew you had to invoke/complete when the instance was ready to serve. It helped that only 4 languages were officially supported.
Integration tests have no standard format and many deployments are combinations of artifacts built from multiple repositories, and configuration is not standardized (configmaps or ENVs? Both? External source of feature flags?) so all integration tests are manual work.
Metrics and SLOs are manual work; only humans can decide what actual properties of the system are meaningful to measure for the overall health of the system beyond simply readiness/liveness checks. Without key metrics automatic rollouts are fragile. This also means autoscaling isn't truly automatic; you need quality load metrics to scale properly. Not all services are CPU or RAM limited, and sometimes the limit varies depending on traffic.
All that said, cloud functions (Google, AWS, or other versions) are beyond DevOps. If you don't need high-QPS services then use cloud functions. They bypass 90% of the headaches of having code running on https endpoints. Most people don't have high-QPS (10K requests per second per shard) services, and could probably get away with cloud functions (1000 RPS on GCP). Everyone else pays the DevOps or hopefully the SRE tax for now. But we're still trying to automate ourselves out of a job; we don't want to be doing DevOps day-to-day either.
by unity1001 on 6/1/22, 1:36 PM
But like everything else in tech, solution of the problems at a given level enables everyone to do much more and build more complex systems that do more complicated stuff on top of that level. We always push the frontier forward with every new solution.
by ciguy on 6/1/22, 5:12 PM
by eric4smith on 6/1/22, 2:37 PM
It’s not.
You still need Devops staff.
Cloud just provisions the hardware and OS. You still have to be responsible for the apps. You still have to be responsible for IO, memory, cpu and networking capacity.
You still need to make sure your apps are able to run on cloud - whether metal or k8s.
by quickthrower2 on 6/4/22, 10:51 AM
The solution to that is PaaS (Platform as a Service), and you can start a startup with almost no devops knowledge using things like Heroku and it’s myriad competitors from startups to AWS offerings.
by hukl on 6/1/22, 2:22 PM
by politician on 6/1/22, 12:41 PM
by dogman144 on 6/1/22, 8:18 PM
by mastazi on 6/1/22, 10:43 PM
Is it? These days I see SAM or Serverless Framework or other FaaS solutions all around me and it seems that everyone is migrating away from ECS/EKS/containers, it might be my own particular bubble though.
by mmcnl on 6/2/22, 5:51 PM
by djohnston on 6/1/22, 1:55 PM
by inportb on 6/1/22, 12:55 PM
by mbrodersen on 6/2/22, 1:58 AM
by fartcannon on 6/2/22, 2:56 AM
Luckily its from the functionally less evil google days and open source so it is possible to use anywhere.
by aristofun on 6/1/22, 5:28 PM
by haspok on 6/1/22, 3:23 PM
by mr_toad on 6/1/22, 10:38 PM
by flyinprogrammer on 6/1/22, 12:55 PM
by pabs3 on 6/2/22, 4:00 AM
by mcharezinski on 6/1/22, 11:41 PM
by lumost on 6/1/22, 2:01 PM
Kubernetes is a powerful tool for "DevOps" roles. It provides an immense array of configuration, and largely replaces many OpenStack, Xen, or VMWare type environments. You can build powerful internal workflows on it to colocate many services on one compute fleet while maintaining developer velocity - which can translate to large margin improvements for some firms. This comes at a cost that you are likely to need a Kubernetes team, and potentially a dev tooling team to make it all work. On a large compute environment, the latter costs don't effect the big picture.
Now on the other hand, more teams than you would expect are just fine on Heroku/AppEngine/AppRunner/Lambda. These teams tend to pay the cost of not having a dedicated dev tooling team through more expensive compute, and sub-optimal tooling. The benefit here though is that "more expensive compute" may mean a fraction of a salary in many environments, and "sub-optimal" tooling may mean a production grade SaaS offering that has a few rough edges you talk to the vendor about.
IME it's much cheaper/lower risk to choose the latter in the long-run. The apparent savings from option 1 eventually turn into tech debt as the shiny tools get old, and migrating to newer/cheaper compute options becomes more expensive. I once built a colo facility which resulted in a 4x reduction in monthly recurring expenses (including salaries) for the same compute footprint, 1 year into the lifetime of the facility the former cloud provider reduced prices by ~30%. Around 6 months into the facility the DataScience team suffered attrition, resulting in fewer compute needs. At the 1.5 year mark the team begged for a flip to SSDs as they were having latency issues (a point of initial contention with the team that SSDs should have been used in the first place). Over the 3 year expected lifespan of the facility there were about ~2.5 months of ramp up/migration work which impacted ROI.
Overall, in hindsight, I'd say at best we achieved a 1.5x reduction in compute expenses compared to the alternative of tooling improvements, cloud cost reductions, and compute optimization. I now seek the tool which provides the lowest friction abstraction as at the worst case I can simply migrate to something cheaper - investing in compute infra has a crazy level of depreciation.
by jrockway on 6/1/22, 2:57 PM
Eventually, people had a hard time finding well-rounded individuals that could design, develop, test, and deploy software. It seems to be a rare skillset, and people are resigned to not being able to hire for that kind of role. So, all of these ancillary concerns got split off into separate teams. You have a design team, a software engineering team, a test engineering team, operations, and so on. DevOps changed from "developers that operate their software" to "developer operations", which is just your 1990s operations team with a new name. You the developer want something, it goes on a backlog for some other team, you wait 6-8 years, you get your thing.
All the complexity of the devops world comes from having one team writing the software and one team running the software. An example are service meshes. They are super popular right now, and everyone and their mother is writing one and selling it for tens of thousands of dollars per year. To the software engineer, having two applications communicate over TLS is pretty simple; you read the certificates and keys from disk or an environment variable, throw them into a tls.Config, and give that tls.Config to your internal servers and internal clients. But, what happens in the real world is that the organization says something like "all apps must use mTLS by January 2023". The software team says "meh, we don't care, we'll get to it when we get to it". So the poor devops team is stuck figuring out some way to make it work. The end result is a Kubernetes admission controller that injects sidecars into every deployment, which provision TLS keys from a central server at application startup time. The sidecars then adjust iptables rules so that all outgoing connections from the original application go through the proxy, and if some distributed policy says that the connection is supposed to be mTLS, it makes that happen. Basically, because nobody on the dev team was willing to spend 15 minutes learning how to make this all work, it got bolted on by $100k worth of consultants, all for a worse result than just typing in a very small number of lines of code by yourself. That's the state of devops. The people writing the software won't run it, so you have to add complexity to get what the organization wants.
I think it's terrible, but that's the fundamental disconnect. When you need to change how software works without being able to edit the code, the workarounds get increasingly complicated.
As always, what looks like a software complexity problem is actually an organizational complexity problem. The way I've managed this in the past is to organize a lot of knowledge sharing, and make a conscious effort to avoid hiring too many specialists. At my current job my team used to make a SaaS product, and our team was a combination of backend software engineers, frontend software engineers, and some engineers with experience with operations. We were one team; frontend engineers would write Go code, backend engineers would make React changes, and we all did operational drills ("game days") once a week. The result was a very well-rounded team. Everyone could deploy to production. Everyone could be on call. Everyone could fix problems outside of their official area of expertise. I wouldn't have it any other way. The industry, however, deeply disagrees with that approach. So you're going to have testing teams, devops teams, etc.
by mountainriver on 6/1/22, 2:46 PM
by kristianp on 6/2/22, 1:16 AM
Economics says that as something becomes cheaper, demand increases.
by broknbottle on 6/1/22, 5:43 PM
by musicale on 6/3/22, 1:02 AM
by imwillofficial on 6/1/22, 1:59 PM
by otabdeveloper4 on 6/1/22, 5:16 PM
by BirAdam on 6/1/22, 1:25 PM
Essentially, these roles have narrowed the focus of system administrators and systems engineers. In one, you are concerned with CI/CD, and in the other you are making and maintaining cloudy solutions for people. This is yet another layer of abstraction for people, but it also means that most people do not know how to configure underlying software anymore. Because they lack knowledge of how to configure underlying software, they also require automation frameworks. They now do not know how to automate their workflows with Bash, Ruby, Python, or anything else. They need the cloud system to do it for them, which means that they get very vendor locked.
EDIT: the plus side of a new abstraction layer is cheaper tech departments at non-tech companies (fewer and cheaper personnel); which also means that pretty much everyone wants to be a software developer now, and very few people want to be SAs, SEs, DOEs, or SREs; you have to know everybit as much but you get paid much less.
All of this may bust. Increasingly, more and more people are becoming wary of monopolistic tech giants. The cost of their datacenters on the planet is increasingly rapidly. The governments of the world are growing wary of their increasing power. For businesses, complete reliance on a third party who has vastly more power isn't as palatable as it used to be. We may see a resurgence of smaller DCs and bare metal deployment, but any such change would only happen if another massive tech bust occurs. The reality that I see is that we may see both models live in tandem indefinitely, as there are differing use-cases that make either more suitable.
by eeZah7Ux on 6/1/22, 5:09 PM
by merb on 6/1/22, 9:17 PM
by janosdebugs on 6/2/22, 6:15 AM
However, there are plenty of companies that will sell you a system, including varying levels of support. You then, of course, have to hire your own DevOps engineers that will deal with the areas the support doesn't cover, which, given the complexity, is still an awful lot. Or you do everything in-house, which means hiring even more people.
TL;DR DevOps engineers won't be out of the job anytime soon. Same for Kubernetes developers.
by fancyfaith on 6/1/22, 4:25 PM
by znpy on 6/1/22, 5:39 PM
by zxcvbnm on 6/1/22, 5:45 PM
by lachlanwhite on 6/1/22, 11:48 PM
by kulikalov on 6/1/22, 12:52 PM
by blodkorv on 6/1/22, 2:34 PM
by throwaway787544 on 6/1/22, 1:47 PM
Back in the day we cobbled together solutions out of different parts because it gave us a strategic advantage over monolithic commercial solutions. It was cheaper, but it was also easy to customize and fit to product & user needs. Yes configuration management was a nightmare, and it came back from the dead as Terraform, because instead of an OS with mutable state we now have a Cloud with mutable state. Docker and Packer and a few other solutions have fixed a lot of the mutable state issues, but databases are still flawed and SaaS is still just a mucky mess of unversioned mutable state and nonstandard uncomposeable poorly documented APIs.
With Kubernetes, we're back in the land of commercial monolithic products. Technically you can build it yourself and customize it all, but it's expensive and time consuming and difficult because of how many components there are tied together. It "gives you everything you need" the way the International Space Station does. Do you need a space station, or a barn?
People get so wrapped up in terminology. Declarative doesn't mean anything other than "not procedural"; it's not better than procedural, it's just different. Plenty of declarative things are a tire fire. Infrastructure as Code just means "there is some code that is committed to git that can manage my servers". A shell script calling AWS CLI is IaC. Doesn't make it a good solution.
You can't just install a piece of software and be done. That's the entire point of the DevOps movement, really. It's not about what you use, it's all about how you use it. Work with humans to figure out what will work for your specific situation. Use your brain. Don't just install some software because it's trendy and hope it will fix all your problems.