10 Years in Testing, 10 lessons – Lesson 6

Hi code coverage lovers, this one is for you.

Lesson 6 – Test Coverage doesn’t tell you much.

I feel like some people in IT get a little too excited when the topic du jour is test coverage. I have never understood this obsession with numbers that tell you so little about how your product is doing.

One time, I worked for a client that had put 80% unit testing coverage as the mandatory number to reach. What was the result? They had to create a “police team” to actually check if teams were producing useful tests, because many teams tried to game this rule by writing true=true as asserts in their unit tests. Good for coverage, good for added bullshit.

Also, what does coverage mean? Function, line, statement, condition, branch, parameter value, loop, state….?

Even if you have “perfect coverage” you also need to have the perfect assertions to go with them because 100% code coverage means nothing if you find unexpected return values (or other side effects).

Even with a tiny function (as shown in the picture above) with guaranteed no side-effects the possible scenarios are huge because of the input values, so how can you get 100% coverage? Well in the real world, there will always be loads of functions working together, so yeah, good luck with your coverage.

Instead of going for high % of test coverage, it’s better to think of what is useful to cover. Remember, we are looking for useful information, not to win some number-game that gives the illusion of control. I do agree with Dan Ashby, that it’s useful to know what parts of your code aren’t covered by tests at all!

But that’s where it stops for me. I’m okay with uncertainty, testing cannot be perfect and risk is part of the game. Identify the biggest risks and focus a lot of your effort there, increase test coverage in those areas if you like, but know that it’ll not guarantee that the software will be without faults.

Thank you for reading, you can find the rest of the blogs here.