Coaching testers on Bug Reports, Advocacy & Credibility

Here are some of the funny/awful bug reports that I have seen in my experience:

• "When I open the application and click on that button I get a error massage saying it fatally crashed"
• "The spelling of Transport Parametre is miss spelled"
• "When I perform a submit the application has error message and thrown on the right side"
• "I open and it crashes"
• "The application is giving me unexpected message to work"
• "Clicking on that link is taking me somewhere and I am unable to return"
• "Applications is throwing pops up when I put my mouse on links of ads"
• "Click fast links and get error message"
• When trying to establish connection with devices some device say I am not available
• Everytime I execute test case TR234 my PC rebooting
• Test case TR 343 fails
• When users click Submit twice & hangs.
• After waking from sleep, application don't respond. (Hope you got what the tester was trying to say)

Impact

• Now, that's funny for the moment but what about the credibility of such testers? 
• Why would they be respected? 
• Why would their bug reports be even read further? 
• Why would they need a hike? 
• Why wouldn't they be treated not on par with others? 

Unfortunately, more number of such bug reports have made developers sick of reading bug reports. That's why you'd see developers call testers to their desk and say "Show me that bug". I am concerned about this problem because it is yours and mine. When we who report bugs in a credible manner join a team and report bugs, the developers still ask, "Show me that bug". That's how bad the effect has been on most developers that I have come across in India. Other country folks reading this could share their experience.


Bug reports are one of the key factors that make or break the credibility for testers.



You know about Hands on Software Testing Training from India? If you haven't, you must take a look at the excerpts of students work. You might be surprised and ask me if fresh college graduates from India really reported such bugs in such a credible way.

Santhosh Tuppad, a tester who chose me as his mentor, striked this distinction - he was profiled at Utest for gaining credibility for the highest bug approval rate. You should read about it, I think.

Santhosh, just has an year of experience. However, the practice he did, in that one year, is what has made him get profiled at utest and win the credibility of the highest bug approval rate. Not just Santhosh, his batch mates are doing exceptional as testers /for their year of experience/ in the organization they work for. That even includes his girl friend :)

I just did what any training is actually supposed to do. So, I am not going to be bragging about myself or about the magic / voodoo stuff of coaching testers. Its safe, you can read it without getting hurt.


Its a shame that some of us are being looked upon as good thinkers for just doing what every tester is supposed to be doing. That's the state in which our community is. I think the difference is that we /those who are considered good thinkers/ are bolder, respect ethics, have higher self respect and want to see the community better.

Let me just share how I coached testers on bug reporting and bug advocacy, so that it could either help you coach yourself or others. The ideas to coach testers that I have is abstracted, borrowed, redesigned, modified from what I observed while getting coached by James Bach & Michael Bolton. I also did a context driven approach to suit India.

Coaching testers to report bugs

There is a one week of training & practice session on Bug Reporting in my Hands on Testing Training class. In typical approaches like ISTQB, CSTE training, they appear to cover it up in 2 hours or less. Wow! Those guys have scalability and I don't.


So, I set up the bug tracking system, provide them a buggy application (mostly any software) and ask them to report all bugs they see. I clearly state that it is not about who finds the most number of bugs but who reports bugs in a credible manner. I don't teach anything about bug reporting when I start this exercise and leave it to them to learn it through the experential approach.



I am logged in as admin from my laptop. The first bug report arrives and I call that participant to my seat. I critique the report in following ways:

The developer view

"I am a developer and I don't understand your bug report. I don't know if this is happening while running in Vista because I see a different behavior in XP. As you have no indication to me about your OS, I have no clue what to do with your report. This is why I don't understand your bug report. I am deleting this report because it doesn't make sense to me. Can you report it again in a way I can understand it?"

• The participant goes back and attempts to report the same bug with additional points based on what I mentioned.
• That participant whispers the message across to other participants and I just act as though I don't know about the whisper.
  

Some kinda smart people pick up a little more and mention other details such as Service Pack 2/3 so that their bug report doesn't get deleted again but others don't get it yet. No problems, Pradeep is there to help.

The next bug from another participant arrives and also gets invited to my seat. "I am a developer who is sitting in US of A and I can't understand your English. Maybe it would help me if you could send me a screenshot or a video. I am going to delete your bug report as I don't find it helpful to me.

•  So, that participant goes back with a little disappointment that although the System configuration was mentioned, the bug report was deleted.
• Whispers about screenshot and videos gets started. I didn't hear any such whisper :)

The next bug from another participant with system configuration and screenshot. A 6 MB BMP attachment."I am working in a dial up connection and this attachment is taking 20 minutes to download. I have lost my interest to see the screenshot. Maybe you logged a great bug but I would have been able to see it if it were a JPG file that takes lesser space"

• Whisper! Sssssh!
• I don't know :)

Another bug, another participant. Spelling and grammatical mistakes, filled and stuffed in the report. "Thanks for coming to my seat when I asked you to come. It shows the respect you have for me. I wish I could respect you that much or maybe even more. However, I see that your English looks so bad that my colleagues would laugh at me if I respect you. In order to prevent that from happening, I have no option than to delete your bug report"

• "Hey, please correct this report" whispers one to another.
• I then announce "Microsoft spelling and Grammar check is your best friend".

As a Developer who loves to say, "No user would do that"


"Hey, the bugs you have reported are not actually bugs. So, I am deleting them all".
"What? Sir, we put in so much of effort and you are deleting it all" (Hate the "Sir" part though)


Well, if you'd not want to me to delete any of your bug report then I need strong evidence and investigation of why you think it is a problem. I wish I knew which Oracle you used but none of you reported any such.

• At this moment, I hear people wondering what oracle they used to spot the problem and trying to add to their existing bug reports.
• I announce, "You could ask me which oracle if you are not sure". For the next 15 minutes I am your friend who knows how to test and the oracles for the bugs you reported.
• Those 15 minutes is real busy time for me.

Next iteration. "The oracles you mentioned seem to be OK but I am still not convinced if that is a problem. It might help me understand the problem if you could tell me the impact it has on the user. Hello! Bug Advocacy! Cem Kaner! Google!"



So that goes on and on and it takes them 3 days to get one bug report to not get deleted by the developer. By then they have surpassed most of the ISTQB, CSTE or any other similar certification training. However this approach that I am talking about here sucks. Man! it doesn't have scale.



As Test Manager


So, from today I am a Test Manager. Not a developer. I hear a sigh of relief. They think their reports wont be deleted as much as it did with the Developer.



"Ah! My manager has called me for a meeting to give an update. When I look into all your bug reports, I get no clue what it means. Your summary is either too long or I am unable to understand them. I don't have time to go through your bug reports in detail. I shall delete all bug reports that do not help me make a quick assessment of the quality of the product"


"Hey! Here is how I report bugs!"

• No Whispers this time.
• They start copying my style and when they are learning it for the first time, that's Okay.
• All bug reports gets changed to a style that I follow.
• The beauty comes when some people try to modify my style to their own style. That's the Indian Masala I like the most.
  

Now, that kind of thing happens for a few iterations and it goes for one full day or so, till I am satisfied about their bug reporting as a test manager.


As a co-learner


"Guys and gals. Now, we are pretty OK with bug reporting and bug advocacy. Let's try to critique the bug reports that are public and try to learn if there is something interesting that others do that we didn't so far"


I open bugzilla.mozilla.com and go through many reports and ask them to point out the good and bad points in the bug report that we are observing. We have a discussion, argument and debates on it. So, we end up refining ourselves and I drive home some important points of bug reporting.


"Folks! Lets test your bug reporting skill. Let's work on another project and see if we repeat any mistakes or make new ones".


Fail fast, Fail Safe


I failed miserably in a couple of exercises that James and Michael did with me as a part of Rapid Software Testing. However, for me, it was safe to fail in front of them than to fail in front of a client. I was glad I was exposed to a context that I failed although that kind of context had not been on my work yet. When the context actually arrived, I said, "Aha! The Wine Glass". Its a RST secret :)


I have a checklist of things I would make the participants fail and get them corrected of it with respect to bug reporting and here are some of them:

• Spelling, Grammar & Typos checks
  
• No usage of SMS language
• Crispy & Useful summary
• Serving different stakeholders
• Observations
• Investigation
• Risk to the user
• Inferences & Conjectures
• Cost versus value
• Screenshots & Videos
• Log files & Supportability
• Symptom versus problem
• Cost of fixing / not fixing the bug
  
• Questions to the developer
  
• System, Browser and all other relevant configuration
• Test Environment awareness & details
• Self critique of ideas and bug reports
  
• Peer review
• Input / Output / In between : test data, system state, environmental changes
• Hardware

Enough! You might ask me how big is my bug report usually. Well, it depends on the bug and the audience not on me. In the last two assignments that I am executing here in Delhi, a light weight bug reporting with all above information is helping me win accolades. Sorry for bragging although I said I won't. Its kinda bad habit, you see.


Oh, you know, I also won the Top Best Bug at Utest in the Bug Battle of Search Engines. I wish Utest could publish that bug report. Since then, I see some testers attaching videos of their bug in utest bug battles and releases. It really solves the problem of language barriers, steps to reproduce for most of the bugs. I didn't invent the idea of video recording bugs but I practice it and advocate it to be used in contexts where it helps.



So, coming back to the topic of Bug Reporting, Bug Advocacy & Credibility. So, are you surprised that Santhosh Tuppad who went through such a training achieved the status of highest bug approval from Utest and is invited to many more releases to test?


Whenever I go and speak to businessmen in India with the results of this kind of training approach and the value it can bring to the community the thing they say after saying "Great" is "This doesn't have as much scale as ISTQB or CSTE where I can make any Tom Sick Hary to replicate the training. Slides are replicate-able. Good luck".


So the message to you is.Forget all this hands on approach and other blah blah. ISTQB has scale. CSTE has scale. CSQA has scale. ISEB has scale.Go there and learn to memorize the CBOK and practice to puke it on the exam.


I don't coach people to memorize. That's not how I was coached. Go find a guru who would teach you good stuff. Don't waste time with businessmen talking about a stuff that they too know is good but they think won't scale.


I will show the world that good stuff can scale. If you didn't like the word "I", let me replace it by another.


"We will show the world that good stuff can scale".



The Avatar!