Notes from spying test experts

This is a top secret post and be very careful with the information. I am not sure if I can expose my notes to my dear blog readers but I am taking a risk of doing it.

There are some secrets of test experts that not many know. I had a mission sometime back to spy about how test experts actually find many important problems, quickly.

Many people think ( that included me, too) that Test Experts do lot more tests and have more fantastic test ideas and that's why they find many important problems, quickly.

I was hoping that there is more information hiding than what is visible about test experts and spying helped to uncover those hidden secrets.

This post is all about the secrets that I found spying test experts and the secrets of how they find many bugs?

You can trust me about the information to an extent that I and many other testers have been practicing the secrets that has resulted to our success of finding many important problems, quickly.


They just do one test to find many bugs!

Now, I am sure you don't want to believe me because my finding says that they do just one test to find many bugs. I am sure you have seen testers finding one bug per tests that find a bug and that makes you not believe this information but... you must understand that the secrets are always unbelievable.

You can feel safe after knowing a fact that I too didn't trust the secret of one test finding many bugs but my spying mission was to find out more granular details about it.

Looking more carefully into it, you might discover that it is not a test that finds a bug but it is a human that finds a bug and a test plays a role in helping the human find it. So there is something with humans that is helping to find bugs and not completely with the tests.

When I collected this much of information over months of spying, I was invited to be a part of the round table conference of experts as I had posed myself as a test expert from India (I knew I wasn't a test expert and even today I am too conscious that I am not one such) but you know... I am a spy and had to pose that way.

Over the round table conference, test experts started discussion about ORACLES* and I didn't know their terminology and just took notes. That evening, when I returned to my hotel, I was trying to connect ORACLES and the spying mission I had in hand.

* Oracles - principle or mechanism by which we (humans) identify problems

I was perhaps lucky that James Bach and Michael Bolton were staying in the same hotel that I was put up and we caught up at the dinner table.

I had an idea that I hope it worked. I was hoping that James Bach is a kind of person who would spill off the secrets in a context to prove me wrong and help me learn the secrets if I talked nonsense about ORACLES, because his passion to the craft of software testing is something that I am trying to match. Michael Bolton, the Birbal of North American software testing was a person whom I had to handle very carefully as he is more likely to identify me as a spy. At the end, I assumed that I was able to outsmart Michael in not letting him know that I am spy.

Going back to the hotel room, I started writing my discoveries about the secrets of test experts and the granular details of how they find important problems,quickly.

Before I publish a report to end the spying mission, I was sure that I had to practice the details I collected. I did practice and was excited that those ideas really worked and I found more bugs by executing one test.

Why many testers might not be able to practice the secrets that I am unlocking here is because they have a fixed expected result that is derived from a specification document that the tester thinks is The Holy Bible for the project.

Specification Document - is one single oracle and hence you might find one bug per test you execute.

The important thing is experts define a bug as "anything that threatens the value of the product" (-- James Bach)

Here are a list of Oracles that experts use with every test they do:

Consistency with the Image: When I execute a test, I try comparing and contrasting it with the image that my company or stake holders have been projecting about the product or the company.

For instance, if my company has an image in the market as people who produce software whose performance is good, and I execute a test whose documented expected result is "This functionality should work", I ask question, "Well it works but didn't it take too long to perform the operation?" and then I find a bug and say, "Although this operation goes through the time it took to complete the operation exceeds beyond the image the market has about us"

Consistency with Similar Product(s): When I execute a test the functionality might work as expected but I ask a question, "Well it works but doesn't it seem to perform in a way that users who are used to similar products might expect it to happen?" and then I find an important problem, quickly.

Consistency within the Product: When I execute a test that appears to have passed, I ask a question, "Well it works but doesn't it work in a different way than other areas of the same product?" and then I find an important problem, quickly.

Consistency with Claims: Marketing personnel and my manager are making certain claims about the product. When I execute a test and the test appears to have passed, I ask a question, "Well it works but does the way it works go against the claim that they make?" and then I find a bug.

Consistency with Statutes: When I execute a test that appears to have passed, I ask a question, "Well it works but does the way it works go against a law that an organization or a certifying authority might have?" and then I find a bug.

Consistency with User's Expectation: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in a way that the users whom I am aware of might not want it to happen?" and then I find a bug.

Consistency with History: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in the way the previous releases of this product work?" and then I find a bug.

While writing this post, I got an e-mail from James and Michael that reads...

Dear Pradeep,

We knew that you were spying us for a long time. We also saw the way in which you took notes, asked questions and drew inferences from what you could capture and that impressed us to help you learn more about testing because those are some important skills that a tester needs to have, to add good value for the cost customers pay for. We consider to hire you and we recommend you to quit spying profession and take up testing where you might get more laurels for your skills. Here is a link to know more about more secrets : www.satisfice.com/rst.pdf

Good luck!

-- James Bach & Michael Bolton

I was excited and I did quit spying, to take up testing as a full time activity. I thought I shouldn't post this in my blog because I didn't want those secrets to be given away for free and saved a draft of this post. I am afraid if Blogger has a bug that publishes drafts that are not intended to be published.


-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton


Update : Michael Bolton, blessed me with a post on his blog for this article of mine. I was jumping in joy for at least half an hour reading that and I am pleased to share it with you: http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html