Photo (c) Janusz Gorycki
Achtung: A Bunch of Obvious Statements Below
Continuing the grand tradition of writing "controversial" posts, let's take a look at unit tests - which for some folks seem to be the new gospel, the best thing since sliced bread, the cure to all our ailing.
Well - the truth is that, in the agile community unit tests seem to be a little bit overrated. Or, to be more precise - they are often incorrectly used as a "silver bullet" substitute for the tedious, labor-intensive, time-consuming, boring and un-gratifying, manual integration and system tests. They quite often, and wrongly, become the one and only QA process in the project. Let me tell you - these hope of getting rid of integration and system testing are false and maintaining them is extremely dangerous.
The Purpose of Unit Testing
Unit tests are a good tool for two purposes:
- they are a fast and simple verification vehicle, allowing you to test your unit's responses to variety of external stimuli - either ones that you could think of while developing, or ones simulating conditions that led to a bug in your code (if these conditions can be reduced to a unit test, which may or may not be the case)
- they give you a cerating assurance that the code you have created for the unit will not be so easily broken in the future by somebody who is refactoring it for whatever reason (bug fixes, improvements, rearchitecting, whatever)
And that is pretty much it.
All other valid reasons for unit testing can eventually be reduced to these two. I remain in a violent disagreement with some of the other "qualities of unit testing" listed in the linked blog post of my esteemed colleague. I won't attempt to go over them one by one here though, for the lack of space.
Don't get me wrong - unit tests are extremely useful - these days they are quite essential. Applying unit test to your code will make it better. You should absolutely be unit testing your code, whenever possible. You should be trying whenever possible to create unit tests mimicking conditions that led to reported bugs. But - it is a criminal offense to stop there. And it is an even bigger crime to insinuate that unit testing alone is a sufficient QA process for any sort of serious software project.
Ok, So How do They "Lie"?
Let's try to answer the question of what purposes unit test are not useful for, or what are the dangers of relying on them, shall we?
- Unit test are not a necessarily a means for documenting your system, sorry. I have seen a fair amount of unit test that did, and an even larger pile of those that didn't, instead increasing chaos and miscommunication. Tests will only become your system's docs, if you create them with this specific purpose in mind. The same can be said about your actual code - it may or may not be self-documenting
- "Green" (all passing) unit tests don't necessarily mean that your system is healthy. It is not humanly possible to test for every scenario, you can break your system in many funky ways and still make your even most stringent unit tests pass. Boy, do I have stories to tell about that one...
- Trusting your unit test mocks is wrong. Mocks are only able to superficially mimick the real world. Claiming that mocks can substitute the real environment is like claiming that a blowup doll is a good approximation of a marriage
- They are worthless if they are slow. In a non-trivially large system, unit testing can take hours (if not more) - and your automated build will take the same amount of time. This makes unit tests worthless as a vehicle for instant feedback - nobody will bother waiting half a day for the results of a build after they committed some refactored code. Yes, you can and should partition your code to smaller modules, but then your builds do not test everything - they only test your module
All of the above are valid problems (and I challenge you to prove me wrong), but the next one is the kicker:
Even if you are a master programmer, you have no idea what to test for. There, I said it. And I have a war story to back me up:
One of the best developers in our team has created a fine piece of code, unit tested the hell out of it and was proud as a baby of his accomplishment. And a fine piece it was indeed - chapeau bas! Yet - it failed "in the field". Its problem was performance - it failed to be efficient in the face of real-life conditions. And the developer in question could't quite believe his eyes whan somebody else on the team investigated the problemand pointed out where exactly in his code the root cause of the problem was - it was a classical case of "this one is obviosly correct, it is not necessary to test it". So if this bloke is unable to create proper unit test suite (and his software development skillz are absolutely top-notch), nobody can.
So please, don't fire your QA department just yet. Their job is still important, even if you unit test.
Bookmark/Search this post with: