by sidpatil on 12/19/23, 2:32 PM with 258 comments
by donatj on 12/19/23, 3:18 PM
I very vividly remember writing a test for a ~40 loc class of pure functions. I started out thinking the exercise was a waste of time. This class is simple, has no mutable state, and should have no reason to change. Why bother testing it?
By the time I was done writing the test I had found three major bugs in that 40 loc, and it was a major aha moment. Truly enlightening.
by corndoge on 12/19/23, 3:26 PM
I still hate writing them and it grates on my aesthetic sense to structure code with consideration to making it testable, but if we want to call ourselves engineers we need to hold ourselves to engineering standards. Bridge builders do not get to skip tests.
by teeray on 12/19/23, 3:03 PM
by agentultra on 12/19/23, 3:33 PM
They are often sufficient for a great deal of projects. If all it takes to convince you it's "good enough," are a handful of examples then that's it. As much as you need and no less.
However I find we programmers tend to be a dogmatic bunch and many of us out there like to cling to our favoured practices and tools. Unit tests aren't the only testing method. Integration tests are fine. Some times testing is not sufficient: you need proof. Static types are great but fast-and-loose reasoning is also useful and so you still need a few tests.
What's important is that we sit down to think about specifying what it means for our programs to be, "correct." Because when someone asks, "is it correct?" You need to as, "with respect to what?" If all you have are some hastily written notes from a bunch of meetings and long-lost whiteboard sessions... then you don't really have an answer. Any behaviour is, "correct," if you haven't specified what it should be.
by the_sleaze9 on 12/19/23, 3:06 PM
I for one do not believe in Unit Tests and try to get LLM tooling to write them for me as much as possible.
Integration Tests however, (which I would argue is what this story is actually praising) are _critical components of professional software. Cypress has been my constant companion and better half these last few years.
by ejb999 on 12/19/23, 3:33 PM
Either I am writing really good code so there are no bugs, or I am really bad a writing unit testing code to find those bugs.
by bicijay on 12/19/23, 3:05 PM
by domano on 12/19/23, 3:17 PM
This takes 3 minutes, 1 if you use tmpfs. It only takes <10 seconds if you dont run writing tests.
These actually cover most real world use cases for a query-engine we maintain.
Unit tests have their place for pieces of code that run based on a well defined spec, but all in all this integration or component-level testing is really what brings me the most value always.
by yitchelle on 12/19/23, 3:28 PM
by mrweasel on 12/19/23, 3:48 PM
If you need to mock out 80% of a system to make your unit test work, then yes, it's potentially pointless. In that case I'd argue that you should consider rewriting the code so that it's more testable in isolation, that will also help you debug more easily.
What I like to do is write tests for anything that's just remotely complex, because it make writing the actual code easier. I can continuously find mistakes by just typing "tox" (or whatever tool you use). Or perhaps the thing I'm trying to write functionality for is buried fairly deep in an application, then it's nice to be reasonably sure about the functionality before testing it in the UI. Unit tests just makes the feedback loop much shorter.
Unlike others I'd argue that MOST projects are suited for unit testing, but there might be some edge cases where they'd provide no value at all.
On caveat is that some developers write pretty nasty unit tests. Their production code is nice and readable, but then they just went nuts in the unit tests and created a horrible unmaintainable mess, I don't get why you'd do that.
by bedobi on 12/19/23, 3:03 PM
in corporate codebases, overwhelmingly, unit tests are just mocked tests that enforce a certain implementation at the class or even individual method/function level and pretend that it works, making it impossible to refactor anything or even fix bugs without breaking tests
such tests are not just useless, they're positively harmful
https://gist.github.com/androidfred/501d276c7dc26a5db09e893b...
by EZ-E on 12/19/23, 3:09 PM
by servaldeneptuno on 12/19/23, 4:11 PM
> Now, what happens when MyThread::singlepassThreadWork() uses a member variable of MyThread like foobar and we delete the MyThread object while the thread is still running? The destruction sequence is such that MyThread is deleted first and after that, the destructor of its parent object Thread runs and the thread is joined. Thus, there is a race condition: We risk accessing the vector foobar in singlepassThreadWork() after it was already deleted. We can fix the user code by explicitly stopping the thread in its destructor
What does it mean when they say 'the destructor of its *parent* object Thread runs'? I've always thought that when you inherit from one class to another and then instantiate an object of said class, they're just one object, so what do they mean when they make the distinction between 'parent' and 'child' object? When you have inheritance of say two classes, those would be two distinct objects instantiated in memory? Is there something I'm missing?
by davnicwil on 12/19/23, 3:22 PM
If any of those conditions doesn't hold the cost/benefit certainly and even sometimes the absolute utility goes way down.
If I have to mock anything, in particular, or more generally care at all about any implementation details (ie side effects) then I just think might as well make this a full on automated functional test then.
As soon as fake code is introduced into the test its utility rapidly decays in time as the things it fakes themselves change.
by m3kw9 on 12/19/23, 3:56 PM
by corethree on 12/19/23, 5:21 PM
You're preaching to the choir. The overwhelming majority of people worship unit tests like dogma. There's almost no point in saying the above. It's like saying it's a little sad to see some people who are so dismissive about eating and breathing to stay alive.
Your next part is the one that's interesting. Mocking 80 percent of a system to get unit tests to work. I've seen so much of this from developers who don't even realize the pointlessness of what theyre doing that it's nuts. They worship test so much that they can't see the nuance and the downside.
Take this article. This article is literally presenting evidence for why unit tests are bad. He literally created an error that would not have existed in the first place we're it not for his tests. Yet he has to spin it in such a strange way to make it support the existing dogma of test test test.
by pfdietz on 12/20/23, 2:36 PM
The former must run quickly, and it's ok if the exact same test is run over and over. The latter need not run quickly, but benefits if new tests can be created and run, or if the tests incorporate randomness so they don't do the same thing each time they are run.
Here, it seems he was using tests intended for the first purpose for the second purpose instead. That can work, as it did here, but I don't think it's optimal. Better to have more exploratory, randomized, property-based tests chugging away in the background to find weird new ways the code can fail.
by cloverich on 12/19/23, 5:58 PM
by baz00 on 12/19/23, 3:10 PM
They confidently broke everything historically and looking forward. Then blamed it on me because it was my test suite that didn’t catch it. The language should not have been broken.
Everything only works if you understand what you are doing so every argument should be posed as both sides.
by JonChesterfield on 12/19/23, 4:30 PM
Whether that's because most software isn't tested competently or because software testing practices don't deliver robust software is not yet clear.
I suspect that unit tests, and tests in general, will be considered a historical artifact from the time before we worked out how to write software properly.
For example, we don't generally unit test things that a static type system checks for us. Maybe good enough type systems will remove the rest of them.
by kreeben on 12/19/23, 9:45 PM
How this topic can sometimes be about belief is beyond me. It's like if a person found a screw driver and says, I now believe in screw drivers.
The topic of how people believe in unit tests, to me is proof that the world is screwed. We're all screwed and everything is a screw driver.
by seanmcdirmid on 12/19/23, 4:17 PM
It would be nice if unit tests were more like interlocking evidence of system correctness, but right now we just have integration tests with poorer coverage for that.
by TheAlchemist on 12/19/23, 9:51 PM
For what it's worth, I find Copilot to be quite an exceptional help in writing unit tests ! A real game changer for me. Not only it takes care on most boilerplate code, but also kind of 'guesses' what case I'm about to write - and sometimes even point me in a direction I would miss otherwise.
by m3kw9 on 12/19/23, 3:52 PM
by HankB99 on 12/20/23, 5:12 PM
It also motivates me to get small pieces working and tested before I get to the finish line. Each successful test is a victory!
by smrtinsert on 12/19/23, 3:49 PM
by pfdietz on 12/19/23, 5:06 PM
by alganet on 12/20/23, 12:53 AM
I'm happy that this article praises unit tests without forcing a TDD perspective to the reader. It presents it like a tool, not a religion, and that's very refreshing.
by drittich on 12/19/23, 4:07 PM
We can argue about what granularity they should be, talk about functional programming, debate whether they should hit the database or not, but IMO all of those things miss the point. For me, in order of priority, unit tests provide the following benefits:
1) Make me write better, more decoupled code
2) Serve as documentation as to the intent of the code, and provide some expected use cases
3) Validate the code works as expected, (especially when "refactoring", which is basically how I write all my code even from the start)
4) Help you when deleting code by exposing unexpected dependencies
You can argue against all of those points, and I often will, myself. It depends on the scale, importance, and lifetime of the project as to whether I will write unit tests. But, as soon as I think someone else will work on the code, I will almost always provide unit tests. In that scenario, they:
- Provide a way to quickly validate setup and installation was correct and the application functions
- Signal that the code was "curated" in some way. Someone cared enough to setup the test environment and write some tests, and that gives me a certain comfort in proceeding to work on the code.
- Provide a gateway into understanding why the application exists, and what some of the implementation details are.
So, thinking about the advantages I've outlined above, for me it would be very hard to say I don't "believe" in unit tests. I just don't always use them.
by onetimeuse92304 on 12/19/23, 3:14 PM
There are multiple problems with unit tests, as they are implemented in the industry. And to make the unit tests usable and productive you need to make them so productive that it can offset those problems.
First of all, for unit tests to work everybody has to contribute quality unit tests. One team member writing unit tests well for his part of functionality is not going to move the needle -- everybody has to do this.
Unfortunately, it is rarely the case that all team members are able to write quality code this is the case for unit tests.
Usually, the reality is that given deadlines and scope, some developers will deprioritize focusing on writing good unit tests to instead deliver what business people do really care about -- functionality. Give it enough time and unit tests can no longer be trusted to perform its job.
Second, it is my opinion that refactoring is extremely important. Being able to take some imperfect code from somebody else and improve it should be an important tool in preventing code rot.
Unfortunately, unit tests tend to calcify existing code making it more expensive to change the functionality. Yes, more, not less expensive. To move a lot of stuff around, change APIs, etc. you will usually invalidate all of the unit tests that work around this code. And fixing those unit tests in my experience takes more effort than refactoring the code itself.
Unit tests are good for catching errors AFTER you have made the error. But my personal workflow is to prevent the errors in the first place. This means reading the code diligently, understanding what it does, figuring out how to refactor code without breaking it. Over the years I invested a lot of effort into this ability to the point where I am not scared to edit large swaths of code without ever running it, and then have everything work correctly on the first try. Unit tests are usually standing in the way.
I think where unit tests shine is small library code, utilities, where things are not really supposed to change much. But on the other hand, if they are not really supposed to change much there also isn't much need to have unit tests...
The most paradoxical thing about unit tests is that teams that can write unit tests well can usually produce code of good enough quality that they have relatively little use of unit tests in the first place.
What I do instead of unit tests? I do unit tests. Yes, you read that correctly.
The trouble with unit tests is that everybody gets the part of what unit is wrong. Unit does not have to mean "a class". Units can be modules or even whole services.
What I do is I test a functionality that matters to the client -- things I would have to renegotiate with the client anyway if I was to ever change it. These tests make sense because once they are written -- they do not need to change even as the functionality behind them is being completely rewritten. These test for what clients really care about and for this they bring a lot of bang for the buck.
by BurningFrog on 12/19/23, 3:45 PM
But in reality, unit testing every single function and method is where the vast majority of the benefit lies. Details really matter.
It took me some time to learn this, even after being told. It's the same for most people. This little post will probably convince no one.
But maybe remember it when you finally get there yourself :)
by dn3500 on 12/19/23, 3:52 PM
https://multicians.org/thvv/threeq.html
I hope he got permission to reproduce the comic.
by hax0ron3 on 12/19/23, 4:02 PM