Your Unit Tests Lie to You

Unit tests will only show flaws in the internal consistency of software, not whether it works outside the programmer's understanding of the unit's purpose. This can make unit tests seem useless, but being more confident of the internal consistency allows one to more readily adapt when the purpose of the unit becomes clearer.

The same is true of types, but types show the absence of flaws, not the presence of them.

I agree with many of the points here, you can't just get rid of QA, you can't just rely on unit tests for everything but they do have their place. I'm always suspicious of anyone thats going to tell me they have one cure for all ills.

So if this bloke is unable to create proper unit test suite (and his software development skillz are absolutely top-notch), nobody can.

Ok so what about the other 99.9999% of times he wrote unit tests, did they suffer from the same problem as well? Are we just talking about an insolated incident or did he always just *miss* things. Many of the bugs we find are because an area doesn't have test coverage or it's something that simply can't be unit tested. We still need QA on this, they aren't going anywhere.

I'm confused. You wrote:

the developer in question could't quite believe his eyes whan somebody else on the team investigated the problemand pointed out where exactly in his code the root cause of the problem was - it was a classical case of "this one is obviosly correct, it is not necessary to test it".

So as described he did think of the case and then decided not to test for it.

By the way, it's true that not you, nor I nor anyone else can think of all the cases for a non-trivial system. We can't for most trivial ones, neither. Which is why we still do need integration testing and QA (although as much of that as possible should be automated: humans should write tests, not do testing). But who's saying otherwise? I've been doing TDD for nearly ten years and intensively unit testing code post-hoc for longer than that and I don't know anyone who's claiming that unit testing removes the need for QA. Przemysław in particular makes no such claim in the posting that you reference.

Ok, can someone please cite the crazy people that think getting rid of integration and system testing is a good thing? I have a hard time thinking that such a thing would even cross someone's mind as more then a sick joke lasting all of a second.

Unit tests are great for the two purposes you mentioned and not so useful in the other ways you mentioned.

However, you didn't mention one of the most important shortcomings of unit tests. Ultimately, what matters is whether the system solves user and stakeholder problems. In other words, it has to fulfill the requirements. You don't improve what you don't measure. So measure the satisfaction of the requirements, both functional and nonfunctional. I.e., use system tests.

Duplication has been removed by Artem. Hopefully correctly
I don't want to comment on that post in general, although I think that underrating (and not using) unit tests is much worse that overrating them). Still, like every software development practice, writing unit tests requires your thought and commons sense.

I don't want to comment on that post in general, although I think that underrating (and not using) unit tests is much worse that overrating them). Still, like every software development practice, writing unit tests requires your thought and commons sense.

With regard to the "war story", I would like to deprive it from its value as I believe I am the guy who Janusz refers to in his blog.
1. the "bug" was about poor performance. Everything except for that was still correct. Just users had to wait too long for the answer. Standard (regular) unit tests are not for testing performance, although unit tests frameworks (like JUnit and its derivatives) directly or indirectly support non-unit testing (like integration testing, UI testing or performance testing)
2. The "fix" was about just changing one regexp pattern
"([/\\\\]?[\\S ]*?([^/\\\\]+\\.java))(:\\[?(\\d+)([\\,:](\\d+)\\]?)?)?"
into another one "^(.*?([^/\\\\]+\\.java))(:\\[?(\\d+)([\\,:](\\d+)\\]?)?)?" and I was not able to find with this change (made by somebody else) any automatic unit test (or performance test) which would:
a) prove that the first pattern is slow and that the second is fast
b) demonstrate (document) which new strings would be matched by this new pattern and were not matched by the old one
3. the code in question was not really well tested, as it was an outcome of overnight programming exercise/tournament called "Fedex" by Atlassian. By design its about having something valuable, innovative and/or eye-candy quickly with less (or no) emphasis put initially on the code quality aspects. If the result of Fedex is interesting, then it's always time for polishing it in regular time (and not at midnight). So your suggestions that this code has anything to do with superb unit tests are just invalid.
Having said that, I still wrote it overnight in TDD way (with unit tests written even before the production code) and even though the functionality is quite powerful and flexible, there have been no single bug report (except for the mentioned here performance problem) since I released it in Atlassian IntelliJ Connector several months ago.

Concluding, I would love to hear pros and cons of unit testing (and real war stories) from more developers who really are expert in unit tests and test-friendly design (i.e. they have been using TDD for many years in various software projects) and know they are talking about.

My 2 cents,

Wojtek

Maybe the Unit Tests aren't lying to you, instead you just aren't listening to what they are really saying. ;)

Unfortunatly I see unit tests and their resulting metrics misused way to much, and I think your blog hit on a lot of the current issues.

For instance, Unit Tests are certainly not meant to replace QA, user acceptance testing, or functional testing. Any development team that believes that they are are most certainly NOT practicing Agile. In fact, the Agile process incorporates product owners directly in the process so that there is continuous user acceptance testing.

Good post.

..when I was lucky enough to be working as a developer, the amount of bugs I would miss when testing my own code. I would always enter data I expected people to enter and therefore my code would handle it. sigh Rob

I know the part about not being able to test all possible scenarios really rings true for me. My company has tested and tested many times, over and over, thinking we had a flawless product only to find out that when it gets in to the hands of users they find "creative" ways to break it. I guess the only solution is more testing but it can be frustrating. Amy

Janus,

I agree with most of what you write. It's just that that the conclusion is misleading.
Unit testing is as good as you write them, and allows you reduce wasted time on manual debugging.
If you don't write them well, yes, you might go chasing your tail. But here's the thing - go practice. You'll write better tests and you'll see through the lies.
I've written a post about metric calculation, and what we thought are the best things to measure, so you can actually tell what works in the process. I'll be happy if you can comment your opinion on that.

Thanks,
Gil Zilberfeld
Typemock