Testing can have many goals: to assess quality, to assess conformance to specification, to help managers decide whether or not to ship, etc. These all affect a test team’s approach. Very often, it seems that “Finding Important Bugs Early” is the tester’s primary goal during a development cycle in an agile environment. It’s the ideal: find a quality impacting issue as quickly as possible after it was introduced so that the code is fresh in developer’s minds. “Finding Important Bugs Early” is a goal shared by countless test teams.
It struck me as odd, then, that James Whittaker of JW on Test in his excellent “The Future of Software Testing” series (Part 4), stated that in the future, we need to “close the gap between when a bug is created and when that same bug is detected.” In my opinion, why wait? There are plenty of things we can do to close that gap right now!
The most efficient tactics I’ve seen are the following: early involvement of testers in the development process, good unit testing, code inspection, static code analysis, continuous integration, and effective use of stubs.
Early involvement of testers is not a new concept; it’s already a part of good test process practices. For example, it is a key area of TPI (Test Process Improvement) known as the “moment of involvement”. As a bonus, I’ve always found that testers make great requirement appraisers; not only can they help assess that a requirement is verifiable, but also that it is complete, consistent and unambiguous. Keeping testers in the loop is especially important in an agile setting where it can help make the documentation barely sufficient, as recommended by this type of development process.
We all know that unit tests are your “first line of defence” for testing executable code. To the developers to whom this task most often falls, I encourage you to ask your test team for help should the need arise. A good test team member has a way of seeing testing from a unique perspective. Use this hidden resource! You can pay the favour back when it’s time for them to use a new automation framework. It’s worth it: a good unit test suite can find many problems early that can be hard to diagnose later downstream.
Code inspection is another early detection tool. Used effectively, it can find logic and design problems at the source, not as a symptom (i.e. when system testing will stumble upon the issue). Moreover, there is no prerequisite of compilation to start it! The same is true for static code analysis. We all know the benefits so apply early, apply often, and save a lot of trouble later on.
For environments that are not of that kind where you “throw the product over the wall” to be tested, continuous integration is necessary. Otherwise, your process may be hindering the test team’s effort to Find Bugs Early. Should your testers be validating last week’s code? Last month’s? If they are, they have no chance of finding your most immediate problems. Don’t forget about bug interactions. It frequently happens that a bug is hidden behind another bug; fix one and possibly expose a slew of others. I heard a question asked once that truly captures the idea: “Why is it that when a bug dies, all the others come to its funeral?” You won’t know who’s in attendance if you’re not testing the latest code.
You may recall that I mentioned the effective use of stubs at the beginning of this post. I hope it piqued your curiosity. It’s something I learned by observing effective testers over the years. When a widget comes off the conveyor belt, they don’t wait for the thingamajig it screws into to start testing. The thingamajig may not be ready for another week! They instead whip up a mock-up of the thingamajig that works somewhat like the real thingamajig and try out their widget, pronto! Very often, the combination doesn’t work as expected and they must ask themselves: are their assumptions about the thingamajig wrong or is it the widget itself? It’s a good thing they didn’t wait until next week to discover this potential problem! This type of experiment enables early integration testing: the component didn’t work correctly in its intended environment. This prompts an investigation into the root cause: is it a bug in the component or a poorly defined interface between two components? Beware however, software stubs can fall into the same trap as a test oracle: you can make it as complex as the real thing if you’re not careful.
These are but a few of the tools for Finding Important Bugs Early. There are many other testing approaches that are not addressed here (so don’t throw away your regression tests!) This post focussed on the “early” part of the goal. What about tactics for the “important” part? Good question! I might be coerced into a follow up post one day…