Code coverage: a tool against unused and untested code
I'm an agile developer because I write my code incrementally i.e. only what I need today. I usually start writing the code by writing Unit Test - it really works and it really helps me understand what should be implemented and how system should behave (I will leave the subject of convincing people, who are new to unit testing, to do "write-test-first" for a separate post). OK - I usually do this but sometimes it's difficult or I'm just too impatient to see the result and start with the implementation and write unit test later. How do I know which packages/classes/methods are tested and which are "not touched" by any single unit test? How do I know that my code is really used somewhere by other parts of the developed system? This is the work for code coverage tool.
What is code coverage
According to Wikipedia article code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing. For me as an agile developer code that is not tested is not used and not needed. I mean it! If the code is not tested and not used it is the code that "will be needed in the future". I don't accept such code - if I need something in the future I will simply add it - if I don't need something at this moment I have to remove it (because not used/tested code may contain bugs).
If some part of my code is not tested it means that I don't need it - that is true in almost 100% of cases. This way code coverage tool shows me not only which parts of the software are tested but also which parts of the system are not used and can be removed.
Example
I use Cobertura code coverage tool that calculates the percentage of code accessed by tests. Let's take a look at this simple project containing 19 classes:
This report does not tell many things - useful information is that line coverage of the whole project is 70% and the branch coverage is 25%. The only conclusion you should get looking at this report is that the project is not well tested (or contain some unnecessary code).
If you navigate deeper and take a look at some classes you could see something like this:
This report tells me as a developer that e.g. getDocs() and setDocs(String) methods are not visited during unit testing. I assume that these methods are not needed thus I remove them (!). You should consider doing the same.
If it occurs that you need those methods write unit test that covers this class and those methods immediately.
Bad example from the past
I have very good example how not to write code and how code coverage tool can help you catching such problems. In the past when I had to write some exception hierarchy I started writing exception class with implementing all constructors from the superclass i.e. java.lang.Exception - just in case ;) Here it is:
This means that I had 4 constructors in each exception class but was using usually only one... Why to write the constructor that nobody wants? There is no need to do it!
Consider following Cobertura report:
This is how I write code now :) I start with the unit test and add the code incrementally only when needed. And Cobertura (code coverage tool) helps me assuring myself I'm using my code.
This way I'm sure at least two things:
1. My code is well tested (maybe not well tested but at least I know to what extent it is tested)
2. I've written only the code that is needed - there is no "dead" code
Conclusion
The best way to write code is to start with unit test and to write it incrementally - implement ONLY what you need today. If you are not feeling good with test-first principle and you prefer write an implementation first (you shouldn't) make sure you use your code. One way to do this is to employ code coverage tool (open-source, or commercial ones like Clover). Such tool will show you which parts of your system need testing or simply which parts of your system are not used and can be removed. After each build I check code coverage report to see how much work I have and I really don't like red color (I love green instead :)
What do you think about usefulness of code coverage tools? How do you write your code - are you really writing your code incrementally? How do you ensure that?
Search
Subscribe to updates
RSS feed
Comments feed
New forum topics
Or Subscribe via email
Comments
We were using a code
We were using a code coverage tool in a legacy application and encountered the following issues:
1. Some components were reused and well tested in a previous application, where they were taken from, so we didn't want to spend much energy on writing the tests for them.
2. Some code was automatically generated for the debugging purposes, it's a madness to write the tests for it and you can't just delete it, because it is very useful when it comes to debugging.
3. I think that an effort of covering a legacy application with the tests is well described with Parreto curve, even worse - some code is just dead and shouldn't be covered.
4. 100% code coverage still doesn't guarantee that your code will be useful, you still may do the wrong thing right.
Overall the results were blurred by the first two issues, the task was difficult because of #3 and probably doesn't make much sense because of #4.
Your post doesn't show how to interpret the results, only the technical details. If I want to integrate code coverage with continuous integration, when I should fail the build? Checking the "red code" manually each time is out of question.
Re: We were using a code
1. Why should you test something that is already tested? If you copy the source code from the legacy application you should also copy the tests? Why? In case you want to modify copied code.
2. No - it doesn't make sense to test generated code - btw. generated code sucks. Why? Try to modify it and then generate again - guess what will happen to your changes :)
3. I still don't get where you are going to? Why would you test stuff that is provided to you by someone else? If it's a crappy code maybe you should consider rewriting it from scratch but I really don't see any value in testing someone elses code.
4. Absolutely right - 100% code coverage does not tell you anything. 100% code coverage should not be the goal of your project. Use your common-sense.
Generally code coverage does not apply to legacy applications/libraries? Why should you test something that is already in place and is probably tested? If you're using some libraries or 3rd party software you shouldn't test it - you're not provider of this software. If you don't trust it to that extent - don't use it.
If you want to integrate code coverage with continuous integration you shouldn't probably set any threshold. Checking the "red code" manually is NOT out of question. Nobody said you have to check it after each build. Code coverage should HELP you identify dangerous fragments of your code. If you make yourself the prison of code coverage tool it's your problem.
Code coverage tool helped me many times in identifying which parts of my project need more testing and which parts can be removed - our experiences could differ - it's quite natural.
Code coverage applies to YOUR code and the code you are writing now.
1. The application was
1. The application was tested, but not with the unit tests.
2. It can work, if you do it smart. For example if you have to support some external API, you can define it as XML and generate the code that verifies parameters, prints logs, etc. and then calls the actual function which won't be generated (imagine a bit disabled implementation of aspect programming).
3. You are right, you should only care about your new code and only measure it for code coverage. But sometimes you have to change the legacy code, should you write the tests then?
I think that without integrating with continuous integration the practice will die in face of some time pressure. I was expecting more geeky answer, like use – a tool that deletes the code not covered by the unit tests :).
Generate continuously, but not necessarily stop the build
My point of view ;)
I think article deals with two topics:
> What is the purpose of TDD (Test Driven Development)?
> What is code coverage?
TDD is a part of XP (eXtreme Programming). By doing your test first, you are fixing the interface of your class and doing the technical requirement (In fact you are playing the role of the customer). This helps you to avoid useless code, make your ideas clear about the responsibility of your class, and the dependency of your class (moke etc.). Thus, it’s the best way to be sure that your code is unitary testable.
Code coverage is a metric… of quality? Of Complexity?
The difficulty to cover a code is directly linked by the cyclomatic number. The biggest cyclomatic number you have, the most exhaustive will be your job. So the code coverage is not a metric of complexity, but it helps to reduce this one.
When you have a good coverage, you can avoid some regression thank to your tests. It helps us to reduce the number of bugs, and thus increase the quality of the code. So this is not a metric of quality, but helps to have a better quality.
My conclusion is that the code coverage is mandatory step to provide good code. Don't forget that tests and code are inseparable. When you re-use a code you must re-use the test...
This was my simple but I hope not simplistic point of view ;)
One more thing
1. My code is well tested (maybe not well tested but at least I know to what extent it is tested)
If you respect the liskov principle or even more you are using a design by contract (cf B. Meyer), your extended class must pass the mother-class' test.
You cannot automate everything
You should consider code coverage tool as a messenger that tells you what's wrong. Code coverage tool will not cause you problems - it will show existing problems in your projects. And these problems were there before but nobody told you about it :)
How you interpret reports is the other story and is very much up to you and the project. I presented my way of looking at Cobertura reports that helped me in many projects. I don't think any automated tool that will delete unused lines of code is cool here - it could remove some critical parts of your system. You cannot automate some processes - most of the time human will decide what to do with the "red code".
And some responses ;)
2. It sounds slippery ;) Generally everything is possible - the question is what is the value of the framework to do round-trip code generation with full bidirectional synchronization, etc.
3. Yes - you should definitely test your code and check whether your changes broke the original code or not.
Code Coverage is a Negative Metric
In other words, the most interesting parts of a Code Coverage report are the red bits. You can be sure that these parts of the code are not tested. The inverse is unfortunately not true for the shiny green lines in the report. I like to consider the percentage of uncovered code in a project, a measure of technical debt that project has.
If I want to integrate code coverage with continuous integration, when I should fail the build? Checking the "red code" manually each time is out of question.
Clover (http://atlassian.com/clover) can optionally fail the build if coverage drops. This is a very good way to introduce code coverage to a team working with a legacy application. Clover also reports which Classes "Moved" between builds. Seeing progress being made towards improving code quality is a good motivator (e..g a live Clover2 report on Guice: http://clover.atlassian.com/browse/guice/historical.html).
@JSabatier A way of thinking about Cyclomatic Complexity is that it is the number of tests needed to really test your code. Reducing Complexity should make increasing Code Coverage, and therefore Code Quality a lot simpler. Check out Clover2's 'Coverage Clouds' which provide a visualisation that combines Complexity with Code Coverage: http://clover.atlassian.com/browse/guice/proj-risks.html
Thanks for the good write-up, Przemyslaw.
Re: Code Coverage is a Negative Metric
Even good tools van be misleading
Nick nailed it! I always had a problem with metrics for software development, but mostly with the way they are used. And this is because they can be manipulated so easily. In an enterprise product usually 90% of the code is trivial and easy to test and 10% or even less is very complex and almost impossible to test in real world scenarios (think distributed computing, multi-threading, limit cases with extremely big amounts of data etc.). A tool like this can provide a false sense of security if it says 92% of our code is tested. While many designers understand the number correctly not the same can be said about management who likes to use it as a proof for correctness and for cutting investments in the infrastructure. You know the classification for lies: lies, damn lies and statistics!
Post new comment