from Hacker News

DevOps Isn't Dead, but It's Not in Great Health Either

by milkglass on 7/6/24, 6:22 AM with 72 comments

  • by surfingdino on 7/6/24, 7:24 AM

    I like DevOps, I don't like being on call 24x7 so I tell my clients I am a software developer not an ops. DevOps was sold as a way to avoid dev and ops teams blaming each other for not shipping software. It was at the time true that devs did not understand hardware and networking and ops did not particularly understand software. We got nice things like Docker and k8s, but the core issues have not gone away, devs don't understand hardware and networking and ops are an almost extinct species. We are re-learning the hard way that certain performance problems are not solved by throwing more hardware at them and that devs will always write Helm charts for convenience and not security. We are also learning that the complexity of networking and backed designs has not gone away we just got more modern tools to describe it. One other unintended negative consequence of misunderstanding DevOps is the proliferation of "full-stack engineers", which is basically an ops, a DBA, a secops, a backend dev, a frontend dev, and a website designer rolled into one. What we have today is a situation pretty much the same as it was before the arrival of DevOps... devs thinking ops are useless, ops thinking dev are no good and ... both thinking the others have no sense of style.

  • by kelnos on 7/6/24, 7:05 AM

    I don't necessarily think this is a bad thing.

    I "enjoy" doing my own ops part of the job because it often feels like I have fewer roadblocks in my way. I build my stuff (or fix my stuff), test it, build some deployable artifact, and deploy it. I do it all on my own schedule, immediately when I'm ready to do each step, and I don't have to wait on someone else.

    But if I'm being honest, I have spent so so so many hours dealing with "ops bullshit", that I probably would have been more productive overall (perhaps worse latency, but better throughput) if I could have just focused on the build/fix and test parts, and kicked the final result over to another team to deal with deployment concerns.

    I'm still not sure which is better. Maybe neither is objectively better, and it depends on the person and organization. Maybe have separate teams, but make your devs do a tour of duty through the ops team every now and then so they understand why the ops people get mad at them from time to time. And vice versa.

    Anyway, I don't buy the CYA/hand-wavy reasons in the article guessing why DevOps is supposedly dying. I suspect it's for the same reason Agile turned out to be a drag on so many developers: people cargo cult and don't really understand why they're doing a particular thing, so they never adapt it properly to their environment.

    So the managers bring in consultants, and the consultants just roll out their cookie cutter solution, with their favorite issue tracker, CI/CD product, deployment system, etc., without really understanding the org's needs.

    Then the managers get upset, and we get articles like this.

  • by siva7 on 7/6/24, 8:49 AM

    DevOps has different meanings to different organizations. Just by skimming over the comments here the problem becomes obvious. For some it is basically a platform integration team that assist product teams for what they call devops, for others they confuse it as a modern sysadmin role and for others like product developers they confuse being on-call "24x7" as some kind of devops principles. It's a heavily abused term, even as we have a common industry definition which is grounded on scientific methods, all of the above examples are still common misconceptions about devops.

  • by InvaderFizz on 7/6/24, 8:23 AM

    The point about multiple deployments per day is well taken. Even those of us who wish to get there, have to contend with decades of process to make headway.

    In my F500, we have a regular release cadence of every two weeks. We would like that to be daily or more, but we are probably one to two years of maturity away from making that a reality.

    Not that we do DevOps in the way it was envisioned. We have a "DevOps" team that is responsible for platforms, processes, and operations, and a developer team that is responsible for the application code. The DevOps team works closely with devs to plan and architect the designs to make sure we're using the right services, scale, tuning, etc.

    Within our org, the only real difference between DevOps and classic Ops is the skill level within the DevOps org is dramatically higher than what you used to find in the sysadmin crowd. Most of our DevOps people are cloud native, but a few have deep systems knowledge and decades of hardware, Linux, networking, security, etc experience. All of our DevOps team is very comfortable with Python, that ends up being our default for glueups. We even have a few former devs who also write custom tooling for our pipelines and other QoL improvements.

    So for us, developers deliver application code, devops does everything else.

  • by ibash on 7/6/24, 7:24 AM

    Performance in the report is defined by how fast you can make a change to production, and how fast you can restore service.

    That’s fine I guess.

    What’s not taken into account is the time wasted on incidental complexity with devops tools.

    For example a highly competent engineer I know is burning hours trying to get ssl working on k8s/gcp with a specific configuration.

    Conceptually ssl is all straightforward, but there’s some arcane knowledge that’s been designed into k8s and gcp. Because they were designed for way more complex use cases… but they’ve also pushed out simpler tools.

  • by digitalnomd on 7/6/24, 1:30 PM

    What I’ve found is that pure “DevOps” roles have become a way for recruiters to sell people on being the ops person without them realizing it.

    I once worked at a place that hired me in as a software dev working in DevOps and the position quickly became consumed by the on-call workload.

    The system we were managing constantly had fires that needed to be put out but we could never fix the underlying problems because someone needed to answer the page which often times was a false alarm.

    So we’d quickly slap on band-aids to band-aids and management gave lip service to fixing the underlying problems with on-call. And the job became about reading the Jenkins build logs because the “devs” thought that was our job. They even forgot how to build and test their own software locally in some cases…

    Needless to say I left pretty quickly and it made me not want to do a pure DevOps job again.

  • by Sparkyte on 7/6/24, 8:14 AM

    It isn't dead, just companies refuse to implement DevOps philosophies because they assume it adds too much friction. When things run some and DevOps appears it isn't doing anything they fire operational engineers. Months later things go to shit because they thought they were not doing anything. Operational engineers or Site Reliability Engineers are essential because they consume toil work which would otherwise be there to distract a developer from developing. No one wants to solve pipelines and delivery methods and ensure stability and deliverability of code. Pipelines need as much continual development as a developer who makes the profitable applications.

  • by sponaugle on 7/6/24, 8:43 AM

    "The report’s authors speculate that “It may be that the ubiquity of DevOps practices has allowed developers and organizations to increase the complexity of projects they are involved in, counteracting the benefits to development velocity. In other words, DevOps practices have likely made the development velocity of complex projects comparable to simpler projects without DevOps practices.”

    So.. in other words - DevOps is in great health.

    As a nit, that would not be "counteracting the benefits to development velocity". It would be increasing it for those complex use cases. It might be lowering the poorly used statistical average, but that is more of a 'you' problem from using exceptionally poor statistical practices.

    I always find these broad 'Something is not dead yet but dying' articles to be of very low overall content value. They far too often depend of either highly anecdotal information, or even worse conclusions based on widespread data averages. Even worse than that, self reported AND self selected data.

  • by JanneVee on 7/6/24, 8:55 AM

    Companies do what they always do. It isn't the first time a organization sees something trendy, say that they adopt it and just renames their old processes to with what is trendy and doesn't change one damn thing. It happened with agile and it is happening with DevOps. These poor performers are most likely doing ops in a traditional way they just adopted the vernacular of modern practices and claiming it doesn't work.

    The core of DevOps and agile is essentially tightening the feedback loops and using the feedback to adjust to become more efficient. "Oh we are failing deploys" and not figuring out why to adjust well that isn't DevOps. Not figuring out why you don't deliver what you promise isn't agile either.

  • by saurik on 7/6/24, 7:48 AM

  • by fulafel on 7/6/24, 8:39 AM

    I think this [1] is mistaking some post-devops developers for "low performers".

    The survey questions in slide deck p 13 (https://cdfound.lfprojects.linuxfoundation.org/wp-content/up...) could all be "no" for someone who deploys to Firebase or a low code platform. Even if they are doing continuous deployment.

    [1] "[...] while 83% of developers are actively engaged in DevOps, there’s been a troubling increase in the proportion of low performers in deployment metrics."

  • by philark on 7/6/24, 12:28 PM

    Senior devops here, the ultimate goal of the devops is to remove itself from the equation, when the cicd gets in a good place we switch focus to other areas like observability

  • by pjmlp on 7/6/24, 12:05 PM

    As someone that besides being a developer, has had build engineer, systems engineer, devops, as 2nd title, this has always been a placeholder for everything else that we do besides coding activities.

    The guy in the team that bothers with setting up computers, physical or virtual, configuring Jenkins, Bamboo, Team City,.., writes the RPM build scripts, and whatever else that other devs rather not do.

    Just like Agile, DevOps and such, they all end up being ways to have conferences, books, consulting services,...

  • by SebFender on 7/6/24, 11:13 AM

    From an outside perspective - I've been in hundreds of meetings around different companies (medium to large size) and these efforts have been deployed in sooo many ways (and interpreted) that it becomes a tough funnel to single out.

    What usually works across companies are somewhat standard systems and flows. But based on what I've seen in the past 2 decades, there's a bunch of different solutions to a similar problem and most have very different ways of solving challenges...

  • by binkethy on 7/6/24, 12:15 PM

    Docker/containers, k8s, and all the bent bloated habits eventually take a toll.

    Sure, it lets you ship that app with postgres, clickhouse, redis, rabbitmq, task runners, workers, the jvm, 5 different versions of nodejs wedged in... But is this really a good thing?

    Its like vacuum sealing poop in unlabeled bags and putting them into someone's refrigerator. Yay, surprise.

  • by okeuro49 on 7/6/24, 11:21 AM

    We have a "DevOps" team, which is ops.

  • by gustavus on 7/6/24, 6:55 AM

    DevOps like Agile was a beautiful idea with a coherent underlying well reasoned philosophy.

    DevOps like Agile was then hacked to pieces and had its corpse paraded around by sleazy consultants and idiotic management that was told "you can save all this money on all your infrastructure people by making your developers do all the ops stuff to." And who doesn't like saving money on those useless crusty old system administrators that don't seem to do anything except whine for "more disk space" if they were so smart they would've prevented that outage la da t week where the logging volume filled up.

    Finally the toolmakers sold the idea that DevOps was something you could buy. Just like the vendors convinced management that buy purchasing Jira you were now agile, now if you have a CI/CD pipeline you are now doing DevOps. But you still can't release without getting approval from the release committee and you can't actually provision a new host in the cloud until you have filled out the requsite approval forms, but key for some reason we uprooted our monolith and threw it on kubenetes so we are doing DevOps.

    It really feels like relieving the Agile cycle all over again in every detail. Just wish I could figure out the next big fad so I could get in on this money printer.

    PS. If you use the term DevSecOps or DevSecFinOps unironiclly your living proof the Dunning-Kruger effect.

  • by mkl95 on 7/6/24, 10:07 AM

    Adopting a DevOps culture is usually very difficult at sales-driven orgs, most of which fallaciously call themselves product-driven orgs. Mostly because PMs see it as a cost rather than an investment, and their sole goal is for devs to ship an incremental ball of crap.

    The only times I've seen DevOps succeed was when it was supported by very senior engineers who were allowed to ignore product people's antics and implement it anyway, and when engineers keen on DevOps inflated their estimates to make room for that kind of work, particularly the initial research and yak shaving.

    Imo, DevOps isn't in great health because the industry isn't in great health, and most non wealthy tech companies are doing panic-driven and FOMO-driven development.

  • by amai on 7/6/24, 2:59 PM

    We will soon have AIOps instead.