Breaking things on purpose
Let's start by introducing a single, simple failure. Change the unit test to expect that adding 1 + 1 will result in 3 instead of 2. That is, change the single statement in the unit test to be: self.failUnlessEqual(1 + 1, 3).
Now when we run the tests, we will get a failure:
kmt@lbox:/dj_projects/marketr$ python manage.py test Creating test database... Creating table auth_permission Creating table auth_group Creating table auth_user Creating table auth_message Creating table django_content_type Creating table django_session Creating table django_site Creating table django_admin_log Installing index for auth.Permission model Installing index for auth.Message model Installing index for admin.LogEntry model ...........................F....... ====================================================================== FAIL: test_basic_addition (survey.tests.SimpleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/dj_projects/marketr/survey/tests.py", line 15, in test_basic_addition self.failUnlessEqual(1 + 1, 3) AssertionError: 2 != 3 ---------------------------------------------------------------------- Ran 35 tests in 2.759s FAILED (failures=1) Destroying test database...
That looks pretty straightforward. The failure has produced a block of output starting with a line of equal signs and then the specifics of the test that has failed. The failing method is identified, as well as the class containing it. There is a Traceback that shows the exact line of code that has generated the failure, and the AssertionError shows details of the cause of the failure.
Notice the line above the equal signs—it contains a bunch of dots and one F. What does that mean? This is a line we overlooked in the earlier test output listings. If you go back and look at them now, you'll see there has always been a line with some number of dots after the last Installing index message. This line is generated as the tests are run, and what is printed depends on the test results. F means a test has failed, dot means a test passed. When there are enough tests that they take a while to run, this real-time progress update can be useful to get a sense of how the run is going while it is in progress.
Finally at the end of the test output, we see FAILED (failures=1) instead of the OK we had seen previously. Any test failures make the overall test run outcome a failure instead of a success.
Next, let's see what a failing doctest looks like. If we restore the unit test back to its original form and change the doctest to expect the Python interpreter to respond True to 1 + 1 == 3, running the tests (restricting the tests to only the survey application this time) will then produce this output:
kmt@lbox:/dj_projects/marketr$ python manage.py test survey Creating test database... Creating table auth_permission Creating table auth_group Creating table auth_user Creating table auth_message Creating table django_content_type Creating table django_session Creating table django_site Creating table django_admin_log Installing index for auth.Permission model Installing index for auth.Message model Installing index for admin.LogEntry model .F ====================================================================== FAIL: Doctest: survey.tests.__test__.doctest ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 2180, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for survey.tests.__test__.doctest File "/dj_projects/marketr/survey/tests.py", line unknown line number, in doctest ---------------------------------------------------------------------- File "/dj_projects/marketr/survey/tests.py", line ?, in survey.tests.__test__.doctest Failed example: 1 + 1 == 3 Expected: True Got: False ---------------------------------------------------------------------- Ran 2 tests in 0.054s FAILED (failures=1) Destroying test database...
The output from the failing doctest is a little more verbose and a bit less straightforward to interpret than the unit test failure. The failing doctest is identified as survey.tests.__test__.doctest—this means the key doctest in the __test__ dictionary defined within the survey/tests.py file. The Traceback portion of the output is not as useful as it was in the unit test case as the AssertionError simply notes that the doctest failed. Fortunately, details of what caused the failure are then provided, and you can see the content of the line that caused the failure, what output was expected, and what output was actually produced by executing the failing line.
Note, though, that the test runner does not pinpoint the line number within tests.py where the failure occurred. It reports unknown line number and line ? in different portions of the output. Is this a general problem with doctests or perhaps a result of the way in which this particular doctest is defined, as part of the __test__ dictionary? We can answer that question by putting a test in the docstring at the top of tests.py. Let's restore the sample doctest to its original state and change the top of the file to look like this:
""" This file demonstrates two different styles of tests (one doctest and one unittest). These will both pass when you run "manage.py test". Replace these with more appropriate tests for your application. >>> 1 + 1 == 3 True """
Then when we run the tests we get:
kmt@lbox:/dj_projects/marketr$ python manage.py test survey Creating test database... Creating table auth_permission Creating table auth_group Creating table auth_user Creating table auth_message Creating table django_content_type Creating table django_session Creating table django_site Creating table django_admin_log Installing index for auth.Permission model Installing index for auth.Message model Installing index for admin.LogEntry model .F. ====================================================================== FAIL: Doctest: survey.tests ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 2180, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for survey.tests File "/dj_projects/marketr/survey/tests.py", line 0, in tests ---------------------------------------------------------------------- File "/dj_projects/marketr/survey/tests.py", line 7, in survey.tests Failed example: 1 + 1 == 3 Expected: True Got: False ---------------------------------------------------------------------- Ran 3 tests in 0.052s FAILED (failures=1) Destroying test database...
Here line numbers are provided. The Traceback portion apparently identifies the line above the line where the docstring containing the failing test line begins (the docstring starts on line 1 while the traceback reports line 0). The detailed failure output identifies the actual line in the file that causes the failure, in this case line 7.
The inability to pinpoint line numbers is thus a side-effect of defining the doctest within the __test__ dictionary. While it doesn't cause much of a problem here, as it is trivial to see what line is causing the problem in our simple test, it's something to keep in mind when writing more substantial doctests to be placed in the __test__ dictionary. If multiple lines in the test are identical and one of them causes a failure, it may be difficult to identify which exact line is causing the problem, as the failure output won't identify the specific line number where the failure occurred.
So far all of the mistakes we have introduced into the sample tests have involved expected output not matching actual results. These are reported as test failures. In addition to test failures, we may sometimes encounter test errors. These are described next.
