by walterclifford on 1/18/21, 7:02 PM with 308 comments
by didibus on 1/18/21, 8:34 PM
The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and headcount you're willing to give, how much tech dept you're willing to take, etc., for the privilege of getting their business.
If you see it for what it really is, you'll see that it works pretty well actually. The business gets more out of you for less to them. It was never about having an accurate timeline or helping with planning or prioritizing, and always about negotiating a better contract with the dev team.
Now keep in mind that the "business" in this case is a person who need to report that through their amazing prowess of administration and management, they personally managed to get X feature out during their last review cycle at Y cost with impact Z. This person will not need to deal with developer satisfaction, retention and performance. They will not need to deal with the impact the lower margins they pushed for had on the next feature delivery, or the continued maintainance of the systems. And if the dev team had to lower the quality too much in order to meet the quote they put out, that will be 100% their fault, the "business" will know not to use them for their next contract, or they'll expect the dev team to take on fixing all the issues at their own expense once more.
by meesles on 1/18/21, 8:47 PM
According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather.
If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.
by jakubp on 1/18/21, 7:32 PM
My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.)
But at that point nobody cares! People go on with further projects, and no data is ever collected.
Nobody learns.
When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away.
Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done.
It's insane.
by csours on 1/18/21, 7:46 PM
Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.
by shihab on 1/18/21, 7:58 PM
Now I know "most papers are pointless" is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research.
[1] I saw Mozilla attempt it, but not sure if it's currently in use.
by Aldipower on 1/18/21, 7:42 PM
There are just to many unknowns you cannot foresee. Software development is complex.
by snidane on 1/18/21, 9:38 PM
The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot "write a function" to deploy cookie cutter houses or bridges one after another.
Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts.
Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance.
Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity.
Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.
by jadams3 on 1/18/21, 7:48 PM
In every single case when you've done that much work, I seem to wind up with a reasonable estimate.
If we've done the job before and have data on it, also reasonable.
Double or triple anything else.
by tootie on 1/18/21, 7:38 PM
by SKILNER on 1/18/21, 8:24 PM
So who are we kidding?
Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team.
About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: "Unsolved Problems of Software Maintenance." You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.
by kylecordes on 1/18/21, 9:22 PM
It's easy to detect this:
Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%?
If you get emotional pushback to discussing uncertainty, they are looking for a promise.
by perl4ever on 1/19/21, 12:52 AM
I found that on average, things take twice as long as expected.
So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it.
A way to look at this is the "unknown unknowns" of anything empirically average to a 100% overshoot.
But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.
by taeric on 1/18/21, 8:38 PM
Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done.
Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?
by mr_tristan on 1/18/21, 8:36 PM
I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, "hey, can I share this?"
So clearly, there's value, at least when we've done them.
I'm amazed at how few places even perform a complete feedback loop. It's just, "when can you get this done?" and, "is it done yet?".
by ChrisMarshallNY on 1/18/21, 8:06 PM
Know how much a Honda Accord costs?
About 25 Grand.
Know how much a Mercedes S450 costs?
About three times as much.
They are both great cars, that will be rewarding to own.
The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better "bang for the buck," although it won't seem like it, on the surface.
The reason is that all those little things that go into high quality take lots of time.
I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot.
However, if you want that app to be ship quality, and highly usable, you're looking at over a month.
The thing is, many folks would consider my test harness lash-ups to be their "shipping" product, and will use that as a basis for estimation.
by encoderer on 1/18/21, 10:04 PM
Don't do an estimate, build a prototype.
Don't do an estimate... seriously... build a prototype.
And then throw it out.
And THEN do an estimate. It will probably be pretty accurate.
Not always easy to do this in a tech company but as an eng director I was able to get it done and it changed a seriously broken process based on multi-week scoping and estimation futility.
by Kaze404 on 1/18/21, 8:27 PM
In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.
by neartheplain on 1/18/21, 7:48 PM
Bit long, but well-presented and worth a listen for any practicing software developer (or person who manages developers).
by BlargMcLarg on 1/18/21, 9:33 PM
Yes, I get on a high level, one needs to be able to say "yes we can make this before [date]", when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with "deploy any time" designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models?
Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.
by Groxx on 1/18/21, 8:54 PM
Everyone wants estimates. For decent reasons. But until we've done a similar thing before, they're utter fabrication. We can take a semi-educated guess, but we know they're still guesses... so for things we don't have concrete answers for, we give small/medium/large/extreme markers for how uncertain we are.
It's fine to take on a big unknown or two. You might even get them both done in a half. But they better be worth it (or you have to decide when to cut your losses), because completing those two could consume all of your resources... and if that happens and you didn't commit everyone to it up-front, you won't get it done this half. Making that tradeoff more explicit managed to get us signed up for fewer low-impact-but-highly-uncertain projects that would inevitably balloon out of control but never be cut.
by rietta on 1/18/21, 11:28 PM
by vincentmarle on 1/18/21, 8:40 PM
Known knowns: familiar tasks that can be reliably estimated from past experience. You can improve these estimates by better knowledge sharing (internal wiki, adding comments).
Unknown knowns: tasks that can be increasingly better estimated the more time you spend on estimating (for example creating Draft PRs with pseudo code).
Known unknowns: tasks that haven’t been done before, so estimations need a decent buffer (30-50%) to account for research and potential blockers. You can improve these estimates by benchmarking your previous efforts combined with the team's skill levels (aka sprint story point velocity).
Unknown unknowns: the unforeseen blockers that come out of nowhere, can't really be estimated, and can really disrupt a project's schedule. Improving these estimates is really hard. The key to improving these is by improving team/company communication and building agile feedback loops so you can identify these issues early and reprioritize as needed.
by gregors on 1/18/21, 8:49 PM
by kfk on 1/18/21, 7:45 PM
by tobyhinloopen on 1/18/21, 10:57 PM
- the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you’ve experience with all parts involved. If you don’t, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final
Now the problem is, this estimate is really expensive, because it’s actual work. It takes about 10-25% of the total project time to estimate the project.
by parentheses on 1/19/21, 2:12 PM
The second key mistake is forgetting that estimates (at least in software) are an upper bound. So high estimates are often “better”.
by ThomPete on 1/18/21, 8:21 PM
When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc)
The problem with that metaphor IMO is that it's not actually what software development is.
The proper metaphor for software development is more "engineering and development of construction site equipment and material" i.e. a developer is not building buildings they are building the things necessary for building the building.
And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does.
I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.
by Netcob on 1/18/21, 8:08 PM
Yet any company that does a lot of the same work could probably use that. You'd still need to do some extra work and tag your issues by things that you want to observe (and maybe predict), but then you could say things like "after we switched to that fancy new API client generator, our client development times went way down but debugging times went up a bit". And the system could display some time range while you're writing the issue. You could also look at the issues at the end of a sprint and then merge them with other items from the time tracker to see how meetings and other distractions played into the outcome.
by gpmcadam on 1/18/21, 8:15 PM
by Londane on 1/18/21, 8:17 PM
by yibg on 1/18/21, 9:26 PM
To borrow the usual building a house analogy, if during software estimation we're actually given how many rooms, how many floors, where are the doors and windows etc, then I think estimates will be a hell of a lot more accurate. Instead, what we typically get is, build a house that can be used by the family of 4, and we'll discover exactly what the family wants as we build.
by grey-area on 1/19/21, 6:51 AM
The broad strokes of the design work need to be done before the estimate (i.e. gather requirements, identify constraints, decide on direction and tools).
If work is unknown or in large chunks, it needs to be broken up into smaller well-understood chunks before estimation.
When new ideas arise, they should usually be kept for future work.
When work is added, other work needs to be removed or the estimate revised.
If the work runs late, something should be removed from the scope to keep it on time, then done after delivery.
If work involves integration with other systems this is much harder unfortunately, but for some classes of software (much business software), it is not so hard to estimate.
by djoldman on 1/18/21, 8:42 PM
>...non-machine learning techniques, e.g., regression.
Is this where we are now? The connotation of "machine learning" doesn't include regression? Wow.
by wyldfire on 1/18/21, 7:45 PM
by morelandjs on 1/19/21, 4:05 AM
You should never claim better precision than you know. It's often more accurate to say "more than an hour but less than a week" than to say "6 hours". We learn scientific precision in middle school... why is modern workflow management still so bad at uncertainty propagation?
by mxcrossb on 1/18/21, 11:09 PM
The article ends by suggesting using GitHub. When I was a PhD student I recall open source projects being a standard data source for people who did software engineering research. But it’s not my field, so I’m not sure if effort estimation really has missed this.
by jacques_chester on 1/18/21, 8:24 PM
by lugu on 1/18/21, 8:48 PM
by galaxyLogic on 1/19/21, 5:28 AM
The time and cost estimates do not measure the amount of technical debt in the outcome. it may good great on paper but that is because of a vaguely defined specification which says nothing about the quality, including amount of te4chnical debt (meaning mostly maintainability).
by franzwong on 1/19/21, 2:40 AM
by domano on 1/18/21, 7:46 PM
by toolslive on 1/19/21, 8:00 AM
(Like "There's a 80% chance we'll finish this today." ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)
by sn_master on 1/19/21, 12:39 AM
by samsk on 1/18/21, 9:22 PM
InternalEstimation = estimate the work as truly as possible (ie. in MD) ExternalEstimation = double the value of InternalEstimation, and increase units by one order
Example: 1 day (MD) = 2 weeks 2 weeks = 4 months
by jugg1es on 1/19/21, 1:34 AM
by k__ on 1/18/21, 8:33 PM
by ChicagoDave on 1/19/21, 5:23 AM
by MattGaiser on 1/18/21, 8:46 PM
by mcguire on 1/18/21, 10:50 PM
by zomars on 1/20/21, 5:56 PM
by deandree_ on 1/18/21, 9:29 PM
by dboreham on 1/18/21, 8:25 PM
Update: working now.
by distalx on 1/18/21, 8:18 PM
by newscracker on 1/19/21, 4:24 AM
by yters on 1/18/21, 8:32 PM