Test errors versus test failures

To see what a test error looks like, let's remove the failing doctest introduced in the previous section and introduce a different kind of mistake into our sample unit test. Let's assume that instead of wanting to test that 1 + 1 equals the literal 2, we want to test that it equals the result of a function, sum_args, that is supposed to return the sum of its arguments. But we're going to make a mistake and forget to import that function. So change self.failUnlessEqual to:

self.failUnlessEqual(1 + 1, sum_args(1, 1))

Now when the tests are run we see:

kmt@lbox:/dj_projects/marketr$ python manage.py test survey 
Creating test database... 
Creating table auth_permission 
Creating table auth_group 
Creating table auth_user 
Creating table auth_message 
Creating table django_content_type 
Creating table django_session 
Creating table django_site 
Creating table django_admin_log 
Installing index for auth.Permission model 
Installing index for auth.Message model 
Installing index for admin.LogEntry model 
E. 
====================================================================== 
ERROR: test_basic_addition (survey.tests.SimpleTest) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "/dj_projects/marketr/survey/tests.py", line 15, in test_basic_addition 
    self.failUnlessEqual(1 + 1, sum_args(1, 1)) 
NameError: global name 'sum_args' is not defined 

---------------------------------------------------------------------- 
Ran 2 tests in 0.041s 

FAILED (errors=1) 
Destroying test database... 

The test runner encountered an exception before it even got to the point where it could compare 1 + 1 to the return value of sum_args, as sum_args was not imported. In this case, the error is in the test itself, but it would still have been reported as an error, not a failure, if the code in sum_args was what caused a problem. Failures mean actual results didn't match what was expected, whereas errors mean some other problem (exception) was encountered during the test run. Errors may imply a mistake in the test itself, but don't necessarily have to imply that.

Note that a similar error made in a doctest is reported as a failure, not an error. For example, we can change the doctest 1 + 1 line to:

>>> 1 + 1 == sum_args(1, 1) 

If we then run the tests, the output will be:

kmt@lbox:/dj_projects/marketr$ python manage.py test survey 
Creating test database... 
Creating table auth_permission 
Creating table auth_group 
Creating table auth_user 
Creating table auth_message 
Creating table django_content_type 
Creating table django_session 
Creating table django_site 
Creating table django_admin_log 
Installing index for auth.Permission model 
Installing index for auth.Message model 
Installing index for admin.LogEntry model 
EF 
====================================================================== 
ERROR: test_basic_addition (survey.tests.SimpleTest) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "/dj_projects/marketr/survey/tests.py", line 15, in test_basic_addition 
    self.failUnlessEqual(1 + 1, sum_args(1, 1)) 
NameError: global name 'sum_args' is not defined 

====================================================================== 
FAIL: Doctest: survey.tests.__test__.doctest 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 2180, in runTest 
    raise self.failureException(self.format_failure(new.getvalue())) 
AssertionError: Failed doctest test for survey.tests.__test__.doctest 
 File "/dj_projects/marketr/survey/tests.py", line unknown line number, in doctest 

---------------------------------------------------------------------- 
File "/dj_projects/marketr/survey/tests.py", line ?, in survey.tests.__test__.doctest 
Failed example: 
    1 + 1 == sum_args(1, 1) 
Exception raised: 
    Traceback (most recent call last): 
      File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 1267, in __run 
        compileflags, 1) in test.globs 
      File "<doctest survey.tests.__test__.doctest[0]>", line 1, in <module> 
        1 + 1 == sum_args(1, 1) 
    NameError: name 'sum_args' is not defined 


---------------------------------------------------------------------- 
Ran 2 tests in 0.044s 

FAILED (failures=1, errors=1) 
Destroying test database... 

Thus, the error versus failure distinction made for unit tests does not necessarily apply to doctests. So, if your tests include doctests, the summary of failure and error counts printed at the end doesn't necessarily reflect how many tests produced unexpected results (unit test failure count) or had some other error (unit test error count). However, in any case, neither failures nor errors are desired. The ultimate goal is to have zero for both, so if the difference between them is a bit fuzzy at times that's not such a big deal. It can be useful though, to understand under what circumstances one is reported instead of the other.

We have now seen how to run tests, and what the results look like for both overall success and a few failures and errors. Next we will examine the various command line options supported by the manage.py test command.