"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Thursday, February 11, 2010

Coaching testers on Bug Reports, Advocacy & Credibility

Here are some of the funny/awful bug reports that I have seen in my experience:
  • "When I open the application and click on that button I get a error massage saying it fatally crashed"
  • "The spelling of Transport Parametre is miss spelled"
  • "When I perform a submit the application has error message and thrown on the right side"
  • "I open and it crashes"
  • "The application is giving me unexpected message to work"
  • "Clicking on that link is taking me somewhere and I am unable to return"
  • "Applications is throwing pops up when I put my mouse on links of ads"
  • "Click fast links and get error message"
  • When trying to establish connection with devices some device say I am not available
  • Everytime I execute test case TR234 my PC rebooting
  • Test case TR 343 fails
  • When users click Submit twice & hangs.
  • After waking from sleep, application don't respond. (Hope you got what the tester was trying to say)
Impact

  • Now, that's funny for the moment but what about the credibility of such testers? 
  • Why would they be respected? 
  • Why would their bug reports be even read further? 
  • Why would they need a hike? 
  • Why wouldn't they be treated not on par with others? 
Unfortunately, more number of such bug reports have made developers sick of reading bug reports. That's why you'd see developers call testers to their desk and say "Show me that bug". I am concerned about this problem because it is yours and mine. When we who report bugs in a credible manner join a team and report bugs, the developers still ask, "Show me that bug". That's how bad the effect has been on most developers that I have come across in India. Other country folks reading this could share their experience.


Bug reports are one of the key factors that make or break the credibility for testers.



You know about Hands on Software Testing Training from India? If you haven't, you must take a look at the excerpts of students work. You might be surprised and ask me if fresh college graduates from India really reported such bugs in such a credible way.

Santhosh Tuppad, a tester who chose me as his mentor, striked this distinction - he was profiled at Utest for gaining credibility for the highest bug approval rate. You should read about it, I think.

Santhosh, just has an year of experience. However, the practice he did, in that one year, is what has made him get profiled at utest and win the credibility of the highest bug approval rate. Not just Santhosh, his batch mates are doing exceptional as testers /for their year of experience/ in the organization they work for. That even includes his girl friend :)

I just did what any training is actually supposed to do. So, I am not going to be bragging about myself or about the magic / voodoo stuff of coaching testers. Its safe, you can read it without getting hurt.


Its a shame that some of us are being looked upon as good thinkers for just doing what every tester is supposed to be doing. That's the state in which our community is. I think the difference is that we /those who are considered good thinkers/ are bolder, respect ethics, have higher self respect and want to see the community better.


Let me just share how I coached testers on bug reporting and bug advocacy, so that it could either help you coach yourself or others. The ideas to coach testers that I have is abstracted, borrowed, redesigned, modified from what I observed while getting coached by James Bach & Michael Bolton. I also did a context driven approach to suit India.

Coaching testers to report bugs


There is a one week of training & practice session on Bug Reporting in my Hands on Testing Training class. In typical approaches like ISTQB, CSTE training, they appear to cover it up in 2 hours or less. Wow! Those guys have scalability and I don't.


So, I set up the bug tracking system, provide them a buggy application (mostly any software) and ask them to report all bugs they see. I clearly state that it is not about who finds the most number of bugs but who reports bugs in a credible manner. I don't teach anything about bug reporting when I start this exercise and leave it to them to learn it through the experential approach.



I am logged in as admin from my laptop. The first bug report arrives and I call that participant to my seat. I critique the report in following ways:

The developer view

"I am a developer and I don't understand your bug report. I don't know if this is happening while running in Vista because I see a different behavior in XP. As you have no indication to me about your OS, I have no clue what to do with your report. This is why I don't understand your bug report. I am deleting this report because it doesn't make sense to me. Can you report it again in a way I can understand it?"
  • The participant goes back and attempts to report the same bug with additional points based on what I mentioned.
  • That participant whispers the message across to other participants and I just act as though I don't know about the whisper.
Some kinda smart people pick up a little more and mention other details such as Service Pack 2/3 so that their bug report doesn't get deleted again but others don't get it yet. No problems, Pradeep is there to help.

The next bug from another participant arrives and also gets invited to my seat. "I am a developer who is sitting in US of A and I can't understand your English. Maybe it would help me if you could send me a screenshot or a video. I am going to delete your bug report as I don't find it helpful to me.
  •  So, that participant goes back with a little disappointment that although the System configuration was mentioned, the bug report was deleted.
  • Whispers about screenshot and videos gets started. I didn't hear any such whisper :)
The next bug from another participant with system configuration and screenshot. A 6 MB BMP attachment."I am working in a dial up connection and this attachment is taking 20 minutes to download. I have lost my interest to see the screenshot. Maybe you logged a great bug but I would have been able to see it if it were a JPG file that takes lesser space"
  • Whisper! Sssssh!
  • I don't know :)
Another bug, another participant. Spelling and grammatical mistakes, filled and stuffed in the report. "Thanks for coming to my seat when I asked you to come. It shows the respect you have for me. I wish I could respect you that much or maybe even more. However, I see that your English looks so bad that my colleagues would laugh at me if I respect you. In order to prevent that from happening, I have no option than to delete your bug report"
  • "Hey, please correct this report" whispers one to another.
  • I then announce "Microsoft spelling and Grammar check is your best friend".

As a Developer who loves to say, "No user would do that"


"Hey, the bugs you have reported are not actually bugs. So, I am deleting them all".
"What? Sir, we put in so much of effort and you are deleting it all" (Hate the "Sir" part though)


Well, if you'd not want to me to delete any of your bug report then I need strong evidence and investigation of why you think it is a problem. I wish I knew which Oracle you used but none of you reported any such.
  • At this moment, I hear people wondering what oracle they used to spot the problem and trying to add to their existing bug reports.
  • I announce, "You could ask me which oracle if you are not sure". For the next 15 minutes I am your friend who knows how to test and the oracles for the bugs you reported.
  • Those 15 minutes is real busy time for me.
Next iteration. "The oracles you mentioned seem to be OK but I am still not convinced if that is a problem. It might help me understand the problem if you could tell me the impact it has on the user. Hello! Bug Advocacy! Cem Kaner! Google!"



So that goes on and on and it takes them 3 days to get one bug report to not get deleted by the developer. By then they have surpassed most of the ISTQB, CSTE or any other similar certification training. However this approach that I am talking about here sucks. Man! it doesn't have scale.



As Test Manager


So, from today I am a Test Manager. Not a developer. I hear a sigh of relief. They think their reports wont be deleted as much as it did with the Developer.



"Ah! My manager has called me for a meeting to give an update. When I look into all your bug reports, I get no clue what it means. Your summary is either too long or I am unable to understand them. I don't have time to go through your bug reports in detail. I shall delete all bug reports that do not help me make a quick assessment of the quality of the product"


"Hey! Here is how I report bugs!"
  • No Whispers this time.
  • They start copying my style and when they are learning it for the first time, that's Okay.
  • All bug reports gets changed to a style that I follow.
  • The beauty comes when some people try to modify my style to their own style. That's the Indian Masala I like the most.
Now, that kind of thing happens for a few iterations and it goes for one full day or so, till I am satisfied about their bug reporting as a test manager.


As a co-learner


"Guys and gals. Now, we are pretty OK with bug reporting and bug advocacy. Let's try to critique the bug reports that are public and try to learn if there is something interesting that others do that we didn't so far"


I open bugzilla.mozilla.com and go through many reports and ask them to point out the good and bad points in the bug report that we are observing. We have a discussion, argument and debates on it. So, we end up refining ourselves and I drive home some important points of bug reporting.


"Folks! Lets test your bug reporting skill. Let's work on another project and see if we repeat any mistakes or make new ones".


Fail fast, Fail Safe


I failed miserably in a couple of exercises that James and Michael did with me as a part of Rapid Software Testing. However, for me, it was safe to fail in front of them than to fail in front of a client. I was glad I was exposed to a context that I failed although that kind of context had not been on my work yet. When the context actually arrived, I said, "Aha! The Wine Glass". Its a RST secret :)


I have a checklist of things I would make the participants fail and get them corrected of it with respect to bug reporting and here are some of them:
  • Spelling, Grammar & Typos checks
  • No usage of SMS language
  • Crispy & Useful summary
  • Serving different stakeholders
  • Observations
  • Investigation
  • Risk to the user
  • Inferences & Conjectures
  • Cost versus value
  • Screenshots & Videos
  • Log files & Supportability
  • Symptom versus problem
  • Cost of fixing / not fixing the bug
  • Questions to the developer
  • System, Browser and all other relevant configuration
  • Test Environment awareness & details
  • Self critique of ideas and bug reports
  • Peer review
  • Input / Output / In between : test data, system state, environmental changes
  • Hardware
Enough! You might ask me how big is my bug report usually. Well, it depends on the bug and the audience not on me. In the last two assignments that I am executing here in Delhi, a light weight bug reporting with all above information is helping me win accolades. Sorry for bragging although I said I won't. Its kinda bad habit, you see.


Oh, you know, I also won the Top Best Bug at Utest in the Bug Battle of Search Engines. I wish Utest could publish that bug report. Since then, I see some testers attaching videos of their bug in utest bug battles and releases. It really solves the problem of language barriers, steps to reproduce for most of the bugs. I didn't invent the idea of video recording bugs but I practice it and advocate it to be used in contexts where it helps.



So, coming back to the topic of Bug Reporting, Bug Advocacy & Credibility. So, are you surprised that Santhosh Tuppad who went through such a training achieved the status of highest bug approval from Utest and is invited to many more releases to test?


Whenever I go and speak to businessmen in India with the results of this kind of training approach and the value it can bring to the community the thing they say after saying "Great" is "This doesn't have as much scale as ISTQB or CSTE where I can make any Tom Sick Hary to replicate the training. Slides are replicate-able. Good luck".


So the message to you is.Forget all this hands on approach and other blah blah. ISTQB has scale. CSTE has scale. CSQA has scale. ISEB has scale.Go there and learn to memorize the CBOK and practice to puke it on the exam.


I don't coach people to memorize. That's not how I was coached. Go find a guru who would teach you good stuff. Don't waste time with businessmen talking about a stuff that they too know is good but they think won't scale.


I will show the world that good stuff can scale. If you didn't like the word "I", let me replace it by another.


"We will show the world that good stuff can scale".



The Avatar!

25 comments:

John Stevenson said...

Wonderful article, very well written. I struggle each day with poor bug reports and the negative feedback I receive from the development teams

Kashif Ali Habib said...

As Cam kaner said:

"The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed".

So bug report should be written in a way to acheive this goal.

Nice post.

Parthi said...

Great Post Pradeep.
This is the BEST blog post I have ever read in recent times, for 2 reasons 'Conviction' & 'Clarity'

I sincerely wish all those organisations (I meant decision makers) who judge people's competency on the number of certifications they complete in a year read this post and understand the value of practical training and application as against "memorize and puke" (Gossh, it stinks)

Thank you for writing a brilliant post.

Unknown said...

great post, Pradeep.
Certainly, one needs to understand the intended audience. and accordingly the test their reports before releasing.
> begin with end in mind (get to users shoes)
> test their work; when they are testing for product quality why not test their own product (test code, documents, bug reports, etc...)
-Ram

sreenadh said...

Good article. Keep it up.

Unknown said...

Very Good Article...

Anonymous said...

the Avatar

lol !!

the self bragging ..moron !

Harsha said...

Awesome

Fake software Tester said...

Hi Pradeep,

Very true. Fake Software Testing is something which is very evidently visible in every cubicle that I see in Indian Companies. And the sad fact is that everybody thinks that they can be a software tester. And a good one at that !!! All of us think that the ability to find a single defect would mean that anyone can test.

Very sadly, most testers that I see want to become developers or testers. And in the Indian IT context, we just do not seem to be respecting a 10 year old software tester.

Pradeep Soundararajan said...

@Anonymous,

the Avatar lol !! the self bragging ..moron !

Thanks. I had two choices, actually. One of being a self bragging moron and other being a coward.

I know what choice I made and discovered what you did.

Ranjini said...

Hats Off again!! You have explained the importance of correct reporting in such a wonderful manner.

Ranjini said...

Hats Off again. You have explained the importance of correct reporting in such a wonderful manner.

Santosh Shukla said...

Hi Pradeep,

Another nice post and very relevant. I have seen many such bug reports in my career. It really irritates me if the member is on my team. :(

The bigger problem is if the very same tester has a big ego and is not ready to accept that he/she does not know one of the most basic task of a tester, Bug Reporting.

Do tell me how I should have dealt in such a situation.

Let there be light!!!

Santosh Shukla
http://proudtester.blogspot.com

Pradeep Soundararajan said...

@ Santhosh Shukla,

I have seen many such bug reports in my career. It really irritates me if the member is on my team. :(

What if they think of you the same way you think of them?

Be careful, we are blind to our own mistakes. Did you consider helping them to report bugs in a credible manner?

I just hope you do that the next time you find such people.

If their ego doesn't permit them to take your help in bettering their bug reporting, ignore and move on.

Santosh Shukla said...

@Pradeep,

Thanks. I understand that when we point a finger at other person, there are four others that point towards us.

I think I do a fair job there.
Yes I tried helping them and some listened.

But many did not and I moved on :(

Santosh Shukla said...

@Pradeep,

Hey Pradeep, My name's spelling is Santosh. Thanks.

Malini said...

Very impressive post!

tina said...

Awesome Awesome & Awesome post...I just regret why I did not find this blog a little earlier...but its never too late.... I can still go thru your archives... Keep posting good stuff pradeep... thanks a bunch!!!

Anonymous said...

This is a blog which plays dirty tricks to get perks.. nothing else.

Pradeep Soundararajan said...

@Anonymous,

This is a blog which plays dirty tricks to get perks.. nothing else.

Sure, and an evidence to the world that great people like you are readers of this blog.

Mukesh said...

Hi Pradeep

Once again a very good post. I liked your idea of video attachments with bug Reports.

Thanks
Mukesh Dhingra

Anonymous said...

15 days since you claimed to scale the "GOOD stuff" ... Still waiting to see some announcements on this? How do you plan to show the world that "good stuff" can scale?

Pradeep Soundararajan said...

@Anonymous,

15 days since you claimed to scale the "GOOD stuff" ... Still waiting to see some announcements on this? How do you plan to show the world that "good stuff" can scale?

I pray God you live for at least a couple more years. However, why are you hiding and want to see the sun?

Fake Software Tester said...

Hi Pradeep,

Nice blog. Inspired me to write a note on how we need to have English as a "Primary Thinking Language" in my blog - http://fakesoftwaretester.blogspot.com.

Please do take a look and let me know of your thoughts!!!

Sunil said...

Hi Pradeep,

That's a wonderful way to teach. Thanks for sharing it.