CIT 591 Unit Testing (Python)
Fall 2009, David Matuszek

Tests and Testability

The most powerful new programming technique--TDD, or Test-Driven Development--results in better, more correct code, and practically guarantees a continuous rate of progress. However, TDD depends heavily on the ability to write and execute tests much more easily than in the past. Therefore, we first discuss the unit testing framework.

Unit Testing

The first unit testing framework, JUnit, was invented by Kent Beck and Erich Gamma in 1997, for testing Java programs. It was so successful that the framework has been reimplemented in every major programming language. Here we discuss the Python version, unittest.

Advantages of testing

First and foremost, of course, a program that has been thoroughly tested will have many fewer bugs. But there are other ways in which testing improves code. A method that has been written with testing in mind:

Is testable. That is, it is possible to state exactly what the method does; what outputs it will produce for any given inputs.
Is likely to be single-purpose. Methods that do several different things are harder to write and harder to debug, but they are also harder to test. Moreover, a method that does more than one thing cannot be reused when you only want to do a subset of those things.
Is less dependent on context--what all the other methods do. If other methods have to be called to set up conditions for a given method to work, the method is harder to test in isolation, and harder to use. By writing tests, you are ensuring that the method can be used in at least two contexts: the test environment, and the application itself.
Uses fewer (or no) global variables. Global variables, though sometimes necessary, are undesirable for a number of reasons. Methods written to be easy to test are more likely to use parameters rather than global variables.
Does no input/output. A method that does no input or output can be reused for other purposes. Input and output should be done by methods with that specific purpose. (It is possible to test methods that do input or output, but this requires different techniques.)
Is easier to modify in the future, with much less chance of unexpected consequences. This makes it much safer to refactor the code (restructure it to make it better, without changing what it does).

The unit testing approach

Unit testing is testing individual "units," or functions, of a program. It does not have a lot to say about system integration--whether the various parts of a program fit together. That's a separate issue.

The goals of a unit testing framework are:

To make it easy to write tests. All a test needs to do is to say that, for this input, the method should give that result. The framework takes care of running the tests.
To make it easy to run tests. Usually this is done by clicking a single button or typing a single keystroke (F5 in IDLE). Ideally, you should be comfortable running tests after every change in the program, however minor.
To make it easy to tell if the tests passed. The framework takes care of reporting results; it either simply indicates that all tests passed, or it provides a detailed list of failures.
- The methods under test should not ask for any input or produce any output -- certainly no output that you need to look at. The testing framework should tell you all you need to know.
- Logic should be separate from input/output in any case.

For a first example, we will use the following code:


def add(x, y):
    return x * y

and save it in a file named arithmetic.py.

We write our tests as a set of funTests are usually put in a separate file, and we'll do that here. We'll call the file testArithmetic.py. Since it is a different file, it will have to import arithmetic in order to access our add method. It will also have to import unittest, Python's unit test framework.

Finally, we need to put our test methods in a class, and tell unittest to run the methods in that class. Because they are in a class, we need to use the variable self in several places. We haven't discussed classes yet, so for the time being, just follow the pattern shown below. Here's the complete structure:


import arithmetic   # import your module (leave off the .py)
import unittest

class TestArithmetic(unittest.TestCase):  #  use a meaningful name

    ## Your test methods go here. Indent your
    ## methods, because they belong inside the class.
    
unittest.main()  # outside the class--this tells the framework to run

Here's an actual test:


def testAdd(self):
    self.assertEqual(4, arithmetic.add(2, 2))

and here are the results of the test:

.
----------------------------------------------------------------------
Ran 1 test in 0.009s

OK

(plus several other lines, which you can ignore)

You probably noticed that there is a mistake in the add method. I "accidentally" picked a pair of numbers that would cause the test to pass (there is one other pair). We can put additional tests within the same method--in fact, it's just a Python method, so we can put arbitrary Python code in it. For example,


def testAdd(self):
    self.assertEqual(4, arithmetic.add(2, 2))      # first test
    self.assertEqual(10, arithmetic.add(3, 7))     # second test
    self.assertEqual(90, arithmetic.add(-10, 100)) # third test

With this code, the "first test" will pass, but the "second test" will fail. The failure will halt execution of this test method (your other test methods will still run), so the "third test" will never be used. The result looks like:


F
======================================================================
FAIL: testAdd (__main__.TestArithmetic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/THUMBDRIVE/PythonPrograms/arithmeticTest.py", line 15, in testAdd
      self.assertEqual(10, arithmetic.add(3, 7))     # second test
      AssertionError: 10 != 21
----------------------------------------------------------------------
Ran 1 test in 0.027s

FAILED (failures=1)

(plus several other lines, which you can ignore)

Notice that the test shows us:

The line number where the error was detected (in testAdd, not in add itself),
The assertion that failed, and
What the actual result was, not just what was expected.

How many tests should you have? You would normally test one "typical" case (for example, 3+7==10 above), and try to test every "extreme" case you can think of. For example, if you were to write and test a function to sort a list, some things you might consider are: What if the list contains some equal numbers? Do the first and last elements get moved to the correct position? Can you sort a 1-element list without getting an error? How about an empty list?

If your test method starts getting too long, or you are testing different aspects of your application method, feel free to create additional test methods.

Here are the rules for writing test methods:

The name of a test method must start with the letters 'test', otherwise it will be ignored. This is so that you can write "helper" methods you can call from your tests, but are not directly called by the test framework.
Every test method must have exactly one parameter, self.
You must put 'self.' in front of every built-in assertion method you call.
The tests must be independent of one another, because they may be run in any order. Do not assume they will be executed in the order they occur in the program.

There are two ways you can call the application methods (the methods being tested):

You can simply import the module (file) with, for example, import arithmetic. To call a method from that module, you must prefix it with the module name, for example, arithmetic.add(2, 2).
As a shorthand, you can say from arithmetic import *. Then you can call the methods directly, as for instance, add(2, 2).

Here are some of the built-in test methods you can call. Each has an optional message parameter, to be printed if the test fails. In most cases, the message is superfluous; don't use it unless you really have something useful to add.

assertEqual(expectedResult, actualResult) assertEqual(expectedResult, actualResult, message): Test that the two values are exactly equal.
assertNotEqual(firstValue secondValue) assertNotEqual(firstValue secondValue, message): Test that the two values are different, and fail if they are equal.
assertAlmostEqual(expectedResult, actualResult) assertAlmostEqual(expectedResult, actualResult, places) assertAlmostEqual(expectedResult, actualResult, places, message): Test that the two numeric values are equal, after rounding to places decimal places (default is 7).
assertAlmostNotEqual(expectedResult, actualResult) assertAlmostNotEqual(expectedResult, actualResult, places) assertAlmostNotEqual(expectedResult, actualResult, places, message): Test that the two numeric values are equal, after rounding to places decimal places (default is 7).
assertTrue(booleanCondition) assertTrue(booleanCondition, message): Test that the booleanCondition is true.
assertFalse(booleanCondition) assertFalse(booleanCondition, message): Test that the booleanCondition is false.
assertRaises(exception, functionName, parameter, ..., parameter): Test that the named function, when called with the given (zero or more) parameters, raises the given exception. Note that, for this assertion method, there is no option for a message parameter.
fail() fail(message): Fail the test. This is useful if you want to create your own tests, more complex that the above methods support.

Dealing with global variables

Recall that the order in which your test methods run is not guaranteed. If one of your tests changes the values of global variables, those changes may or may not have been performed when some other test is run. In order to test successfully, you need to restore all global variables to their initial state before each test. This is where the setUp(self) method comes in.

If you write a method setUp(self), that method will be called before each and every test. Like this:

list = a list of all your test methods
for testMethod in list:
    call setUp()
    call testMethod

Test Driven Development

Test-driven development takes a simple but counterintuitive approach: Test the code before you write it. In pseudo-Python, we can describe the process as:


  while the method isn't complete:
      write a test for the desired feature
      run all tests and make sure the new one fails
      while any test fails:
          add/fix just enough code to try to pass the tests
      refactor the code to make it cleaner

Except for very simple methods, you can usually write the method in stages. Try to take small steps. Write a test for the simplest case you can think of, then run it. The test should fail--if it doesn't, then either the method already handles that case, or you did something wrong in the test. This is a quick way to make sure you aren't wasting your time adding code to the method.

Next, write just enough code to make the test pass. The temptation may be strong to write more code than necessary, but resist. Don't write any code that you don't already have a test for. Make sure the new code passes, or debug it until it does.

At this point your method may not do everything it needs to, but it will be correct in the part that it does. Now is a good time to think briefly about how to refactor (make cleaner) the code. Is there a variable that could have a better name? Would it be more readable if you rearranged the parts of an if statement? Could you pull out a chunk of the code and make it a function? Did you repeat code? Refactoring is all about making the code better without changing what it does.

Advantages of TDD

Writing the tests first has advantages over and above simply having a good set of tests. By writing the tests first:

The methods are guaranteed to be testable.
You develop a clearer understanding of just what the method should do (thus avoiding the "Ready, fire, aim!" approach to programming).
You are more likely to write single-purpose methods.
You are more likely to write input/output free methods.
Your methods will be less dependent on context (since it would take you additional work to create that context for testing purposes).
You can trust any methods your new method calls, because they will already have been tested thoroughly.
You will write and test smaller amounts of code at a time. Since it is much easier to test a few lines of code than it is to test a complete program, this means you will make steadier progress toward completion.
You will likely have a more thorough and complete set of tests.