from Hacker News

Ask HN: Best books on managing software complexity?

by zenkat on 2/6/22, 1:16 AM with 124 comments

What are the best books you've read on how to manage software complexity, both from an architectural as well as organizational perspective?

  • by mdaniel on 2/6/22, 3:16 AM

    I enjoyed "A philosophy of software design" by John K Ousterhout https://www.worldcat.org/title/philosophy-of-software-design... just be forewarned that at 180 pages, it's a lot more aspirational than instructive. It depends on what kind of organizational problem you're having: lack of experience, lack of interest, lack of discipline, lack of buy-in, or $other

  • by kqr on 2/6/22, 7:51 AM

    The least complex software is the one never built. We often over-automate. Automation is brittle and inflexible and easily ends up lined with edge cases and other types o complexity.

    Automation is great when it replaces a stable, well-working manual process.

    The way to introduce automation is to first experiment with humans doing something manually until you have a great process. Then take the dumbest, most reliable part of that and turn it into a computer program that alerts a human if it encounters an edge case.

    Try not to handle edge cases in the software, but try to remove them from the greater system in which they occur.

    Automate by elimination, not by complexity.

    Good books in this vein would be anything with Taiichi Ohno in the title, or, perhaps better, as the author.

  • by kqr on 2/6/22, 6:51 AM

    I feel like a broken record for the amount of comments I've posted related to Deming and lean product development these past few days, but as Taiichi Ohno says, the biggest waste is overproduction.

    A lot of the things we build into our software is complexity nobody really needs. One way of managing it is by never making it in the first place. This takes at least two things:

    - Thorough discussion with the customer about what problem they really are trying to solve. They will come to you with what they think is the solution. That way leads unnecessary complexity.

    - Rough economical analysis of choices. Often we do things because that's what one ought to do. Make a quick napkin calculations. A lot of the things one ought to do are not actually necessary.

    There are way too many books to recommend here, but perhaps some of the most relevant are

    - Deming's The New Economics, and Out of the Crisis,

    - Reinertsen's Principles of Product Development Flow,

    - Ward's Lean Product and Process Development.

  • by zenkat on 2/6/22, 5:42 PM

    OP here. Thank you for all of the great suggestions! I've compiled all the references into this spreadsheet: <https://docs.google.com/spreadsheets/d/1kgFSWp7rM3ZegweXAoMN...>

    From my perspective, the top "must-reads" are:

    -- John K Ousterhout, A Philosophy of Software Design

    -- Titus Winter (et al), Software Engineering at Google

    -- Hanson and Sussman, Software Design for Flexibility

    Other interesting titles that caught my eye:

    -- Peter Naur, Programming as Theory Building

    -- Scott Wlaschin, Domain Modeling Made Functional

    -- Yehonathan Sharvit, Data-Oriented Programming: Unlearning objects (upcoming)

    -- Nick Tune, Patterns, Principles, and Practises of Domain Driven Design

    -- Robert L. Glass, Facts and Fallacies of Software Engineering

    -- Donald Reinertsen, The Principles of Product development Flow

    -- Eric Normand, Grokking Simplicity

  • by jupp0r on 2/6/22, 2:36 AM

    “Software Engineering at Google” has been eye opening for me. It hits what you are looking for in its first chapter and then goes into details of different aspects in later chapters.

    https://abseil.io/resources/swe-book

  • by blowski on 2/6/22, 9:05 AM

    I agree with much of what's already said here.

    1. Figure out the difference between your inherent and accidental complexity. The Cynefin framework can be a useful way of thinking about what types of complexity you have.

    2. Attempt to reduce (or remove) the complexity at root before assuming you must model it. "The biggest tragedy in business is doing well that which should not be done at all." For example, we had an important user still on a legacy browser because they couldn't afford to upgrade their old laptop. Instead of adding a complicated legacy layer, we bought them a new laptop.

    3. Think of the whole system, rather than just shifting the burden somewhere else within the system. This is why microservice architectures so often fail - engineers think they've reduced complexity because each service is so simple. Yet all they've done is move the complexity into the operational layer, where it's harder to deal with. Gerald Weinberg is a fun writer in this space.

    4. In many organisations, a lot of people's salaries depend on their ability to add complexity, especially to processes but also to code. If you are working within such an organisation, you will have a Sysiphian task until you solve the organisational problem. Whether you can solve this problem depends on how much influence you have. This is why "skunkworks" projects so often work until they are re-integrated into the mainstream.

    5. Once you've sorted out the organisational complexity, separated the inherent and accidental complexity, you'll be most of the way there. Keep reviewing and challenging everything. Complexity is a weed that just seems to grow unless you constantly tend to the garden.

  • by pluggles on 2/6/22, 3:16 AM

    A philosophy of software design by John Ousterhout https://smile.amazon.com/dp/1732102201/ref=cm_sw_r_apan_glt_...

    The entire focus of the book is reducing complexity.

  • by adamddev1 on 2/6/22, 7:22 AM

    This might seem more basic/beginner level but I really have enjoyed working through How To Design Programs. [1] It teaches you how to think about data, types, abstraction, and creating mini languages/sets of functions for robust, adjustable, and maintainable code. It's helped me immensely as I've started to build much more complicated stuff. It's amazing how you can follow the design recipies, break problems into parts, plug functions together, everything just works. Applying these principles in a typed language (TypeScript) has massively improved my ability to much more complex stuff that is also more safe and maintainable.

    [1]: https://htdp.org/

  • by Twisol on 2/6/22, 3:21 AM

    Peter Naur's "Programming as Theory Building" is a classic essay, and absolutely worth reading. This transcription [1] (with some extra remarks at the end by the transcriptor?) is pretty accessible.

    [1] https://pablo.rauzy.name/dev/naur1985programming.pdf

  • by froh on 2/6/22, 9:26 AM

    At an organizational level, Donald Reinertsen "the principles of product development flow" is outstanding.

    https://books.google.de/books/about/The_Principles_of_Produc...

    It motivates small batch sizes, decision making at ground level, short cycle times, all the mechanics behind kanban, scrum, XP, without ever using any of their jargon.

    And it moves up into economics by explaining e.g. cost of delay as a metric guiding useful decision making.

    The book eatablishes a wonderful shared understanding of the principles underlying lean and agile, why they work and how to make them work, shared understanding between engineering and management.

  • by bear8642 on 2/6/22, 2:35 AM

    Yet to experience proper work life, but really enjoyed Brook's The Mythical Man-Month

  • by MoritzWall on 2/6/22, 3:07 AM

    Architectural perspective:

    Clean Architecture: A Craftsman’s Guide to Software Structure and Design https://www.oreilly.com/library/view/clean-architecture-a/97...

    Organizational perspective:

    An Elegant Puzzle: Systems of Engineering Management by Will Larson https://press.stripe.com/an-elegant-puzzle

  • by deltaonefour on 2/6/22, 9:28 AM

    Written by the authors of sicp: https://mitpress.mit.edu/books/software-design-flexibility

    Organization is just a name for a group of methodologies. Ultimately what we want is the right way to organize programs for maximum flexibility.

  • by fred123 on 2/6/22, 9:00 AM

    Out of the tar pit is the best paper on this I’ve ever read: http://curtclifton.net/papers/MoseleyMarks06a.pdf

  • by rramadass on 2/6/22, 12:51 PM

    I think it is worthwhile to first look into the fields of Complexity Science/Complex Systems Science and Systems Thinking/Systems Theory for overarching insights : [ https://en.wikipedia.org/wiki/Complex_system and https://complexityexplained.github.io/ ] and [ https://en.wikipedia.org/wiki/Systems_thinking and https://thesystemsthinker.com/systems-thinking-what-why-when... ]

    For Software specific issues i highly recommend David Parnas' collected papers in the book Software Fundamentals. He and his colleagues defined much of what is mainstream in today's Software Architecture/Organization practice.

    Finally, the case studies given here are a great source of insights: https://aosabook.org/en/index.html

  • by yuppie_scum on 2/6/22, 1:29 AM

    These are more architecture focused but maybe relevant.

    The Phoenix Project. The DevOps Handbook. The Google SRE Book. And “Kill It With Fire.”

  • by ncfausti on 2/6/22, 9:41 AM

    I’ve been liking Grokking Simplicity so far. It starts out by separating code into three parts: data, computations, side-effects.

    You want to minimize side-effects, preferring computations over them, and data over computations.

    I’m still early into it, however, this already has been a helpful lens to view my code through.

    https://www.manning.com/books/grokking-simplicity

  • by triebelf on 2/6/22, 9:35 AM

    I found this one very good: A Philosophy of Software Design

    https://www.amazon.com/dp/173210221X

  • by kqr on 2/6/22, 6:43 AM

    Michael Feather's book on legacy code contains a lot of advice that ought to be obvious, and yet every time I'm handed a scary piece of complex legacy software, I leaf through Feather's book and I almost always find something useful.

  • by zonovar on 2/6/22, 9:41 AM

  • by jnmandal on 2/6/22, 4:00 AM

    Domain Driven Design by Eric Evans

  • by throwaway2037 on 2/6/22, 4:59 AM

    "Large-Scale C++ Volume I: Process and Architecture" by John Lakos

  • by eterps on 2/6/22, 7:19 AM

    Domain Modeling Made Functional - Scott Wlaschin

  • by bradleysmith on 2/7/22, 4:31 AM

    Not a book, but this paper “big ball of mud” has resonated with me in describing troubles I’ve faced before, and has me coming back to it as I run into new architectural problems where I’m not confident in my/our approach.

    http://www.laputan.org/mud/

  • by tq-qv on 2/7/22, 10:48 AM

  • by sedeki on 2/6/22, 4:22 PM

    This is a book that focuses on managing software complexity and to "avoid painting oneself into a corner":

    https://mitpress.mit.edu/books/software-design-flexibility

  • by Ozzie_osman on 2/6/22, 3:25 PM

    - Philosophy of Software Design (great short read packed with insight)

    - Mythical Man-Month (the classic, a little outdated but the principles still ring very true)

    - Software Engineering at Google (thoughts from a company that has spent a lot of time thinking about this at scale)

  • by andreskytt on 2/6/22, 9:23 AM

    https://link.springer.com/chapter/10.1007/978-981-15-5856-6_... Has a bunch of useful references. Olivier de Weck of the MIT held an awesome conference speech at the 2016 MIT SDM Conference explaining sources, metrics and cost of organizational complexity linked to engineering complexity. Sadly, the video no longer appears online. But the work of him and his group (references in the article above) is highly relevant to your question.

  • by rTX5CMRXIfFG on 2/6/22, 5:44 AM

    I haven't read any of the books on architecture recommended in this thread, but I learned how to manage software complexity by:

    1. Counter-intuitively, being deeply interested and reading about a topic in natural science. Software is complex because it is a system, and especially if you're using a programming language with plenty of OOP features, you're going to benefit a lot from understanding how nature separates properties and functions into organic matter, how they work in concert to achieve a particular effect, and how they compose each other to scale up from the atomic to the cosmic scale; or

    2. If you can't be bothered to learn about topics beyond programming itself, then practice unit testing. As a matter fact, attempt to write unit tests for your old code. If you can't make the unit test run, then that's life slapping you in the face to tell you that your code is way too tightly-coupled.

    By the end of the day, doing any of the above things (but more so, both!) will teach you the true essence of KISS.

  • by irthomasthomas on 2/6/22, 1:27 PM

  • by kennethh on 2/8/22, 8:40 AM

    Surprised nobody have mentioned Making software, https://www.amazon.com/Making-Software-Really-Works-Believe/...

    It takes a empirical view on the software process. Does writing tests first help you develop better code faster? Can code metrics predict the number of bugs in a piece of software? Do design patterns actually make better software?

  • by KingOfCoders on 2/6/22, 12:40 PM

    First keep it simple, shameless self plug: https://www.radicalsimpli.city/

  • by david_allison on 2/6/22, 12:02 PM

    Michael C. Feathers - Working Effectively with Legacy Code

    Scott Wlaschin - Domain Modeling Made Functional: Tackle Software Complexity with Domain-Driven Design and F#

  • by fulafel on 2/6/22, 10:56 AM

    Do you want to find out about managing a bad case of complexity in an existing project? Or are you searching for ways to avoid getting into a bad place with sw complexity and looking for even radical simplicity because you are determinde to avoid (or can't afford) complexity?

    These are quite different missions since it's rarely possible to recover form complexity if you've already gone there.

  • by sakesun on 2/6/22, 9:27 AM

    For me it's Design Patterns: Elements of Reusable Object-Oriented Software

    The essay in the the beginning of the book gave me a profound epiphany back then.

  • by m0rc on 2/6/22, 1:22 PM

    As a baseline on what is know and not know about managing software projects, I always recommend Robert L. Glass Facts and Fallacies of Software Engineering.

    More speculatively, I think that the following biology readings are inspiring:

    - Gerald Jay Sussman, Building Robust Systems an essay. In general, biology should be a source of inspiration for engineers.

    - E. Goldratt, The Goal

  • by kqr on 2/6/22, 7:06 AM

    Software complexity is often made worse by organisational complexity. Someone put it as "make sure people do complex jobs in simple organisations rather than the other way around.

    David Marquet's books have been surprisingly helpful to me in shaping simpler organisations, starting, of course, with Turn the Ship Around.

  • by mikevm on 2/6/22, 11:01 AM

    I suggest looking into: https://www.amazon.com/Righting-Software-Juval-L%C3%B6wy/dp/...

    It's very unorthodox, but cuts through a lot of the bullshit.

  • by bobm_kite9 on 2/6/22, 9:17 AM

    I touch on both these perspectives in my book ‘Risk-First Software Development’. You can read all it online at https://riskfirst.org if you want.

    Hope this is useful to you.

  • by AussieWog93 on 2/6/22, 11:14 AM

    At least from where I'm at (ex-SWE, quit at a relatively junior level), I'm not sure this is something you can read about and reason logically. It's something you need to a feel for through experience.

  • by rrrodia on 2/6/22, 1:26 PM

    It's a long shot, but is anyone aware of useful MOOCs that cover this topic?

  • by aristofun on 2/6/22, 7:59 AM

    Grady Booch “OO Analysis & Design”

    Classics

  • by glintik on 2/6/22, 3:30 PM

    Best book is task description how to add small feature to your project. If it’s still clear and easy, then you don’t need to read any books about software complexity management ).

  • by bkgh on 2/6/22, 2:53 PM

    Object Thinking by David West