This is the first post in a two-part series about the importance of measuring efficient functional tests in platform engineering and in complex products. Baubak Gandomi is a Test Architect with Adobe Customer Journey Management.
Code coverage is often a KPI used to show progress in projects. Especially when the quality of a legacy product needs to be brought up to speed. It is often expected to help detect holes in our certainty regarding the infallibility of our products and their associated tests. We think that although it seems quite clear in its results, it is quite misleading if used on its own to represent the efficiency of your tests.
This is actually a very good measure and it allows us to find lines in the code that are not tested. But I feel that it does not give a clear picture of the state of the tests, and leads to false assumptions when you generalize this for tests that are not unit tests.
The main focus of this article is functional testing. We will often use the term “line-level” coverage to designate coverage reports that point to a line of code. This includes among other things, line- and branch coverage.
Problems with Line Level Coverage and Functional Tests
Although we think line level coverage is very well adapted to low deployment tests such as unit and some component tests, we think it has a few drawbacks when using it in relation to functional tests. In this chapter, we will give reasons.