Great Bug Reports
Is your team maintaining a bug report dump? When new issues get reported at a pace the team cannot keep up with it’s in trouble. At some point the team must yield and bug reports will start to pile up. The unattended bug reports lose their value over time and become a terrible waste.
Shallow Testing
There are many possible reasons for a bug tracking system to turn into a dump of wasted effort. One reason is when the testing team is practicing shallow testing. That is when the testers file bug reports right after discovering bugs, leaving any further investigation to the developing team. This is convenient for the testers but bad for the overall process of finding and fixing bugs.
The shallow testing technique results in many bug reports, most of them with poor quality. But the measure of effective testing is not how many bug reports it produces. What counts is the amount of bug reports that lead to bug fixes. To be effective we need great bug reports.
Great Bug Reports
So, what does a great bug report look like? We need to ask another question to answer that. What does the developer need to fix a bug? The first step of a successful bug elimination is to reproduce it in a controlled development environment.
In order to do that he needs to:
- Put the system in a state from which the error can be produced.
- Perform the actions that produce the error.
- Know how to identify the error.
Most bug reports contain enough information to make step 2 and 3 clear. Clues toward step 1, on the other hand, are usually missing. As a result the developers end up doing additional testing to pin down the problems.
If we do a better job with our bug reports we can get two really important effects. We help developers spend less time on each bug and instead fix more of them, and we increase the chance that the report will be acted upon. This is why thorough testing on behalf of the testing team is so important.
Focus On the Right Things
The basic formula to produce great bug reports is as follows.
- A descriptive name.
The most important details of the bug presented in a sentence. - Steps to reproduce.
This is the part that distinguishes great bug reports from lesser ones. A well-described sequence of action steps that describes how to reproduce the bug removes uncertainties and signals thorough research. Developers are therefore more inclined to act upon such a bug report than a fuzzy one.
It’s also important that the reproduction steps start at a known state, preferably at a clean program start. - Expected results.
- Actual results.
- Any possibly relevant attachments, for example screen-shots and data.
Depending on the system you’re using, a multitude of additional parameters may be set to categorize the bug further. Provide the extra information if applicable, but it’s not essential so don’t let it steal focus from the basic formula.
Bug Mining
The foundation of a great bug report is built with great testing. Great testing is hard work and requires a special mindset: the love of bugs and the willingness to spend a lot of time examining each new specimen.
When a great tester discovers a bug the first thing she does is not to fire up the error reporting tool and start describing it. She leaves that for later. Instead she goes about to find the answers to these two questions:
- Can I reproduce this bug?
- If so, what is the smallest number of steps I need to do so?
This usually requires several program resets, attacking the bug from different angles. I like to call this process Bug Mining and it’s the tester’s most important task.
Why all the hard work?
Why should the tester have to do all this work, you might ask. After all, the developers introduced the bugs in the first place so they should clean up their own mess, right? Besides, isn’t it better if the testers spend their time testing and finding more bugs?
The problem is this: If the testers prioritize quantity over quality we’ll end up with a very ineffective test-correction-test cycle, and an ever growing mountain of unresolved bugs.
The more time developers spend on each bug the fewer bugs get corrected. And if fewer bugs get fixed, more time will pass between bug discovery and bug correction, making it even more expensive to fix.
Since the developers are the only ones that can make the bugs go away you really need to think twice before making them do work someone else can do just as well. This is especially true for Bug mining.
Guiding Principles
As a summary, here are some guiding principles to produce great bug reports:
- Focus your effort on the steps to reproduce the bug. They should be clear, complete and as short as possible.
- Only one bug per report.
Handling several bugs as one is problematic for several reasons. For example, what should you do if the bug report contains bugs with different severity? Also, it’s difficult to describe several bugs in the name sentence so some might be overlooked. A third reason is the communication problem that happens when only some of the reported bugs get corrected. - A few really good bug reports is better than plenty of not so good ones. Quality is better than quantity when it comes to testing.
Cheers!