Wednesday, August 19, 2009

Low code coverage doesn't always mean low test confidence

Code coverage is not always the best indicator of a test suite's usefulness, expressiveness, etc.

Contrary to the popular belief, there are cases in which high test coverage is not a good measure of test quality. In fact, high test coverage is not always possible.

Consider the following situation:

public class Container {
object a_1;
object b_1;
object c_1;

object z_1;

public object A_1{
return a_1;
a_1 = value;


public class A {
Container container;

public A (Container c)
container = c;

/*all methods of class A interact with container.A_1 ONLY*/


The other lettered classes follow the same format.
When Pex tests the Container class it will more than likely get 100% test coverage.
When Pex tests the lettered classes, however, it will not get 100% code coverage. The test coverage report includes coverage of the Container class, which does not require 100% coverage to test the lettered classes. The classes could arguably be written so that they are more easily testable; however, I did not write this code. I actually observed code similar to this in a real application. The lettered classes get 100% block coverage themselves but overall block coverage is never 100%.

1 comment:

  1. "lettered" means "latter"?

    Why would the latter classes not get full coverage?