Monday, 13 July 2015

The test first attitude

Problem statement:

I want to ensure a number is between the range of 5 and 9, inclusive.

Thinking in Test first. 

I can produce 5 test cases for this very simple problem.

A test of a number less than 5. 2. Returns false
A test of the left bound. 5. Returns true
A test of a number in the middle of the range. 7. Returns true.
A test of the right bound. 9. Returns true
A test of a number greater than 9. 15. returns false

Taking a step back, we want to be really thorough we could write the following tests as well.
A test just outside the left bound. 4. False
A test just inside the left bound. 6. True

A test just inside the right bound. 8. True
A test just outside the right bound. 10. False

I define an interface to the method.

public boolean ensureNumberIsIn5To9Range(int numberToTest);

First I code up the 9 tests against the specified interface.

Execute all tests. They fail.

Now I start the solution code... I'm done when all tests pass.

I come up with a pretty standard if statement, using logical And and two return statements. The tests all pass.

However now I realise I could make this more concise and prettier for the reader. So I refactor the code to just use logical AND and simply return what it evaluated to the caller. Tests all pass.

Happy days.

Now I check the code coverage, just to be sure it meets targets. 100%. I'm done. Check the whole lot in.

Thinking in solution first approach

I'm going to promise to do some tests to get us code coverage, with a code coverage metric target of 80%. Management love code with a coverage of 80%.

I will just code up a if statement, with two boundary clauses anded together. Simple. Will then write a test that triggers the boundary clause. And another test that doesn't trigger the boundary condition so happy out.

Next I'll have a think about the interface, it's gonna be pretty simple in this case.

Open up my IDE. Fire in the interface, the algorithm and now I'll think about tests.

So looking at my code, I can see the branch (if statement). The first test I write, just takes any number outside the boundary condition. So I trigger the method with 15, expecting false. I run it, its passes and I get 75% code coverage. Wow I'm nearly there with just one test!

If I trigger the boundary condition on the if statement, then I can increase this figure. So I'll write one more test. This time I'm going to pass 7. Right in the middle of the boundary condition, I am really expecting it to pass.

I execute it. It passes. 100%!!!!!!! Happy days. Check the whole lot in. Home time.

But wait...
Did you spot the bug in the solution first approach? The upper boundary has been incorrectly coded... values of 10 will return true. It's a good thing this code wasn't used to control an auto mobile safety system, an aircraft or a train!

So I've got 100% coverage, but I'm still leaking bugs. Why?

The set of useful unit tests

We explored the set of useful tests in another article. In this simple example we see the set of useful tests consists of 9 tests. I should implement these 9 tests to ensure correctness of the code. The set of tests that gives 100% code coverage is just 2 tests. This is a significant minority of the tests I actually need to ensure correctness. Hence when reality throws in something we didn't test for, we find bugs, although I have 100% code coverage. You can see there is a large scope for bugs, even in this simple application when you test for coverage.

Spending time writing more automated tests around your top 20% of all uses cases that your users use, will give you a much greater bug count reduction in future releases. Spending any time increasing code coverage when you haven't got that 20% of your code base well tested, is waste.

1 comment: