The tester who tested his testers

Hi Reader,

This is an interesting story that came alive during a heated discussion with my manager Srinivasa. The discussion was on "bias" that multimedia testers are prone to while testing a product. (I shall talk about that in detail, in some other post but for the time being the focus is on the the story.)

Its story time again in Tester Tested!

There were two multimedia products under test, lets say Bino and Nino. A tester was testing both of them. Bino was started earlier than Nino and had reached a state where the subjective quality of the product was good enough. Bino on reaching a good enough state was made the benchmark and Nino's quality was tested against it. Bino was talked all over the company for its quality.

Each time the tester tested a release for Nino, he used to report Bino was still better than Nino. Of course, mulitimedia quality is all about subjective view but as testers we need to quantify; why we rate something poor.

For some releases, Nino was not as good as Bino and hence the testers view was acceptable. After a few such iterations, the manager felt Nino's quality had improved a lot but the tester kept rating Bino to be much better than Nino.

The manager performed a trick to find out "what is going wrong with the testers decision despite the quality seems to have improved?"

He labelled Bino as Nino and Nino as Bino and gave a release of Nino to the tester.

This time the tester gave a report saying "Bino is still better than Nino" which means the tester was biased to Bino and had formed a pre-concieved notion that Bino is the benchmark.

Now, that was a great story about "bias of a tester" and the risk of "bias" on a tester. The manager was none other than my current manager Srinivasa.

Now, some of you might form an opinion that my manager was doing a micro management. Actually if you look closer, my manager helped the tester to come out of the bias.

Now my manager has set an example to other test managers as to how they can help their testers to look into themselves. I am sure if your manager gets to read this, he would have something to learn.

This story reminds me of one of the lesson I learnt from James Bach - As a good tester, you should never say "I am sure" since what you say is what you have observed/conjecture/infer but the truth could be different.

As a tester, you do not know whether you are being tested by the product or manager and it is recommended to say "I conjecture/infer/looks like Bino is still better than Nino".


Thanks and Regards,

Pradeep Soundararajan
pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

Disclaimer: Do not get biased by the title of this post. It is not to mean that test managers test their testers, it is a style of expression I used to make the learning a joy.