Comments on: why i broke 89 tests

By: Caffeine Driven Development » Blog Archive » L33t Links #35

Caffeine Driven Development » Blog Archive » L33t Links #35 — Fri, 30 Oct 2009 07:35:22 +0000

[...] why i broke 89 tests [...]

By: Troy Frever

Troy Frever — Thu, 29 Oct 2009 20:03:49 +0000

I agree with your general approach. In fact, the problem is not that 89 tests broke. People often complain about that type of thing, but in truth it is not the issue. What can be problematic is if it is difficult to find your code error among the failing tests. If you were able to immediately say, “aha, the problem is in SimulatedPolymorphism” then it doesn’t matter how many other tests broke.
Your plan to write a unit test when this happens is one good way to solve this. Another is to try to always make sure some integration tests hit every class in a meaningful way. If you change a class and 200 tests fail, but one of the failures is in a suite named for the class you just changed, then you immediately know where to look.
Of course, if your integration tests run slowly, that is a different issue. I am assuming that yours run as quickly as typical unit tests, in which case I find the extra test coverage to be a bonus, not a smell. Again, given that you have solved the issue of rapid location of the underlying error.

Troy Frever

By: Kent Beck

Kent Beck — Thu, 29 Oct 2009 14:40:25 +0000

Thank you for the writeup. It’s always instructive to dig deeper into a situation that surprised you.

I examined the results of running 200M tests over 50 person years and found a clear power law distribution in the number of failures per test run. Sounds like you hit the fat tail (or the fat tail hit you).

I agree with your analysis of the dilemma–readability vs. jackpot failures. Like you, I’ll take readability every time. 40% of the tests for JUnit are at the API level and we don’t regret it, in spite of the occasional multiple failure.

By: Kevin Rutherford

Kevin Rutherford — Mon, 19 Oct 2009 19:08:37 +0000

Yes, I test-drove a fix for SimulatedPolymorphism and then set about writing this up.

I do think it’s always worth paying attention to the little nagging voice whenever I’m developing, because that voice is almost always telling me something important. When I can figure out what it’s saying…

By: Marty Andrews

Marty Andrews — Mon, 19 Oct 2009 00:42:49 +0000

FWIW, I have the same attitude, and the same problem, in the Roodi code base. The integration tests are much more expressive, and seem to be where I focus now that the code is reasonably stable.

By: Robert Lewis

Robert Lewis — Sun, 18 Oct 2009 22:44:37 +0000

So, once you fix SimulatedPolymorphism your 86 tests will pass again, right?

I don’t think this is anything to get too worked up about. I know people complain a lot about how fragile integration tests can be, but if the tests are valid and maintainable they are valuable checks on the behavior of the system. If you have a reliable integration environment, they can be very quick to write as well.

I would tend to segregate them from more focused unit tests, because they tell you different things. It sounds like you’re approaching this the same way I would — write a focused unit test that shows the problem, then fix it. All your tests pass again, and life is good!