Sunday, September 21, 2008

100% code coverage - good but not preeminent

I was configuring TestDriven.NET on my machine, and the idea of writing a few lines about code coverage popped in my mind. There have been continuous debates and discussions about this particular topic in the past years and thousands of words have already been written by experts in the field. Everyone can have a different opinion about testing, and more importantly code coverage (as this is what the post is all about), and so do I.

We, programmers (irrespective of the programming language that we use), have one common goal in our respective objectives - to write good quality code. And by good quality code, I mean code that implements correctly the desired functionality, is in line with industry (or corporate) standards while following the appropriate coding guidelines, uses constraint resources of the environment and infrastructures in an optimal way and is easily maintainable, just to mention a few. Last but not least, another feature of the good quality code is also 'bug free' code.

The testing phase in a software development life cycle ensures that the code is well-tested using an appropriate strategy. Different types of tests, such as unit tests, integration tests and system tests, just to name a few, are conducted. Once the tests are completed, code coverage reports are usually produced.

"Testing never proves the absence of faults, it only shows their presence." - Edsger Dijkstra

What is code coverage? Code coverage tells us how much of our code have we tested. It gives helpful information about the part of the code that has been exercised, and hence, a complete new perspective to the testing paradigms. However, the toughest task is to be able to correctly interpret the code coverage results (or say, reports). And that's where development teams and programmers often make their first mistake.

Many professionals in the software development field believe that a 100% code coverage means that the application is bug free. Hold on. Take a moment and give this a second thought. Are you indeed such a believer?

According to me, 100% code coverage is good but not preeminent. Not preeminent? Yes, I believe there is nothing extraordinary with a 100% code coverage. Does the 100% coverage guarantee that all the tests have been done in the right way? Does it guarantee that the selected test cases and test data are indeed correct? Does it guarantee that we haven't missed any corner case? What about a missing functionality in the code, will the 100% coverage identify it? The answers are a big NO. This is neither the Test Driven Development nor the code coverage fault. It's a failing of the way the results are interpreted. The focus should be more on the test strategies, scenarios and data rather than on the code coverage.

Let me give a very basic example. I have a static Divide() method in the Divider class that accepts a dividend and divisor and returns the quotient.

using System;
using System.Collections.Generic;
using System.Text;
 
namespace CodeCoverageTests
{
    public class Divider
    {
        public static double Divide(int divident, int divisor)
        {
            return dividend / divisor;
        }
    }
}


I will use the DividerTest class to test this Divide() method using the NUnit tool.

using System;
using System.Collections.Generic;
using System.Text;
using NUnit.Framework;
 
namespace CodeCoverageTests
{
    [TestFixture]
    public class DividerTest
    {
        [Test]
        public void TestDivide()
        {
            Assert.AreEqual(10.0, Divider.Divide(30, 3), "Incorrect division.");
        }
    }
}

When run, the test is passed successfully.


And more importantly, the code coverage shows a 100% figure.


With this figure, it gives the impression that this is really a good piece of work, with tests passed succesfully, and above all, a code coverage of 100%.

Ladies and Gentlemen: Don't be fooled by the coverage report. That's how Andrew Glover expressed himself in his article “In pursuit of code quality: Don’t be fooled by the coverage report”.

Call the method Divide() with 0 as the divisor and the program will throw a DivisionByZero exception. This is just a simple example to show how wrong one can be when assuming that a piece of code with 100% code coverage is bug free.

Code coverage is a very good indicator of how various sections of the code were tested. It is here to work as a complement, not the primarily indicator, for the overall code quality picture.

No comments: