Viewing code coverage (Advanced)

We have gone over a lot of PHPUnit functionality and how to use this functionality to build a test suite. Once you have a test suite created it is good to understand how effective your unit tests are. One way you can measure the effectiveness of your tests is by seeing how much of your code is actually under test. This is commonly referred to as your test suite's code coverage. PHPUnit has very useful reporting tools to help measure and monitor this coverage.

Getting ready

To generate code coverage reports you must have XDebug installed. This can typically be accomplished using your operating system's packaging system.

How to do it...

Make the following modifications to the phpunit.xml.dist file as shown by the highlighted lines of code:

<phpunit
    bootstrap="test-bootstrap.php"
    colors="false"
    strict="true"
>
  <!-- other content -->
  <logging>
    <log type="coverage-html" target="build/html-coverage" charset="UTF-8" highlight="true" />
  </logging>
  <filter>
    <whitelist processUncoveredFilesFromWhitelist="true">
      <directory suffix=".php">src</directory>
    </whitelist>
  </filter>
</phpunit>

Now any test runs will also generate an HTML code coverage report for your code.

How it works...

There are a couple of new PHPUnit configuration options presented here. The first and more important one is done via the <logging> and child <log> element. We are instructing PHPUnit to use the coverage-HTML logger type. You must specify a target attribute. This can be either a relative or absolute path to a directory that the HTML will be written to. If the directory does not already exist, it will be created. You can also specify the charset attribute that should be used to generate the HTML as well as whether or not the PHP source code will have syntax highlighting.

The second configuration you can use is the <filter> element. This is used to specify a white list, a black list, or both of files that you want to be included in your code coverage report. Generally, you should use a white list with the processUncoveredFilesFromWhitelist attribute set to true. This will make sure that all files in that directory are included in the coverage report. Otherwise, only files that are actually loaded will be added to the report. This means you run the risk of files that have 0 percent coverage falling off of the coverage report completely. This will give you a false confidence in the coverage of your tests.

The contents of your white list should be the code for the system you are testing. Usually, there is not much value to your testing to be gained by including the third-party code or the test code itself. These filters can be used to ensure this third party and test code is not considered in your coverage reports.

The <whitelist> element can include one or more of the <directory> and <file> elements. These elements contain absolute or relative paths to the directory or file that should be included. The <directory> element also accepts a suffix attribute to further filter the contents by the contained files' suffix. In our case, we used .php. In addition to these two elements, you can also include an <exclude> element. This element also takes one or more <directory> or <file> elements. This can be used if there are specific files or directories that are in white listed directories but should be excluded. It should be noted that you can use wildcards (*) in any of these elements.

If for some reason you would prefer to use a black list, then you simply need to use the <blacklist> element instead. There are no attributes for <blacklist> and you would use the same child elements as you would use with <whitelist>.

Now, when the phpunit command is run, an HTML version of the coverage report will be written to book/build/html-coverage/index.html.

This displays all of the files in our src directory along with a summary of the test coverage in that file. There are three different types of code coverage that are broken down for the directory as a whole as well as each individual file.

The line coverage is the first measurement. It is a percentage of the executable lines of code that are executed by the test suite. The function or method coverage is the percentage of methods or functions in the file/directory that have full code coverage. Full code coverage means that every executable line in the method or function has been executed at least once. The final metric, classes and traits coverage, is the percentage of classes or traits in a file or directory that have full code coverage.

From this view you can drill into your directories. This would show a view similar to what you've already seen. If you click on a file however, you start seeing some very useful information.

This view presents a summary of the coverage of a specific file. It will include a count of the classes in the file as well as how many of those classes have full coverage. It will then break down each method or function in the file and report whether or not that function has full coverage, and how many lines in each method is covered.

Each method also gives a Change Risk Anti-Patterns (CRAP) index. CRAP is a good metric for identifying methods that are hesitant to change. It is a function that ascertains the complexity of a method and the amount of coverage the method currently has. All other things equal, the more complex a method the higher is its CRAP index. The index is lowered as you create more tests that cover more of a method.

There is no hard-and-fast method for determining the optimal CRAP index. So I will leave the details for that debate to other books and papers. I prefer to see methods with CRAP indexes less than 10.

Once you get below the summary table you will see a syntax highlighted rendering of the source code for the file. Each executable line will be highlighted either in red or green. If it is red, that means it is not covered by any tests in your suite. If it is green, that means the line is covered by one or more tests. If you hover over a green line, it will show a list of all the tests that cover that specific line. You may occasionally see lines highlighted yellow. These are dead lines. This means that due to the structure of the code, the yellow highlighted line could never possibly be executed. This is usually because of a return statement somewhere other than the end of your method or function.

PHPUnit also provides a dashboard view of your project at book/build/html-coverage/index.dashboard.html.

This dashboard provides four interesting pieces of information. The class coverage distribution shows you how many classes in the system fall within a specified range of code coverage. In an ideal project only the right-most bar would have any size. That would mean that all of your classes have full code coverage. This project is problematic as you can see that a large number of classes have absolutely no coverage.

The second section is a scatter plot of the class complexity. The Y-axis measures complexity and the X-axis measures code coverage. The ideal code has lots of coverage and little complexity, so the ideal location for your plots is going to be in the lower-right corner of this graph. The worst possible location for any plot is the upper-left corner. If you see a plot in that area, you can find the class that is the culprit by hovering over the plot.

The third section shows your top project risks. This gives you a more comprehensible view of what you see in the class complexity chart. You can use it to identify where your testing effort should be focused. The classes here are determined by using the CRAP index we already covered.

The final section is the least tested methods section. This gives you an overview of the methods with the least amount of testing focus. This drills down even further on areas that can be improved in your test cases.

There is a wealth of information that can be a great help in improving an existing test suite's coverage. It can also go great lengths in keeping the quality of an already good test suite high.