When 100% code coverage is not always enough

As part of our build process we run several hundred unit tests. Once these have completed execution, we then run code coverage analysis. This gives us a raw figure of the percentage of the code that has been exercised by the unit tests. Currently this is running at over 90% code coverage.

Even if we had 100% code coverage, this doesn’t mean the code is immunue to faults. Whilst having 100% code coverage is a good figure to aim for, it doesn’t imply that your unit tests have tested your entire codebase. How can this be? Surely having 100% code coverage means you have exercised every line of code? In fact this is where obsession over code coverage can lead to an over confidence in your testing strategy.

Here’s a simple example.

Hide Copy Code

int counter = GetNewCounterValue();
 
if (counter == 0)
{
    //do something here
}

In the example above, we can easily write a single unit test that will exercise all lines of code. We just ensure that when we arrange our unit test we inject a value of zero into the test harness. By doing so, our unit test will enter the if condition and exercise all lines of code. But what about the implicit else condition. Shouldn’t we test that also? The answer is of course, yes we should. So we also need to write another unit test that injects a non-zero value into the test harness. So although our first test exercised all lines of code and therefore gave us 100% code coverage, we needed two tests to give us full conditional (branch) coverage.

This is where using code coverage alone can be a blunt tool. It is a useful indicator, and can be used to measure relative code coverage between different parts of the code. For example, it can be useful to see where your unit tests are weak, and where they are strong (relative to each other). But code coverage shouldn’t be used as an absolute value on its own. In isolation it is pretty meaningless. It’s real value comes when used to give comparative measurements of code coverage throughout the codebase.

It’s also important to know the critical areas of the code, and ensure that these areas have adequate testing coverage. For example, it’s probably important that your login functionality is adequately tested, as this is critical to the security of the application. So you probably want to invest more time and effort in ensuring that these critical areas of the code are tested more thoroughly than other lesser critical areas of the code. Not all areas of the code are equal. So not all tests are equal either.

So whilst it’s important to have unit tests, it’s also important to ensure that you spend time ensuring that all branches of the code are covered (not just the lines of code), and that the more critical areas of code have adequate testing coverage relative to other lesser areas of the code.

If this article was helpful to you, please do hit the 💚 button below. Thanks!