"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Wednesday, September 19, 2007

Notes from spying test experts


This is a top secret post and be very careful with the information. I am not sure if I can expose my notes to my dear blog readers but I am taking a risk of doing it.

There are some secrets of test experts that not many know. I had a mission sometime back to spy about how test experts actually find many important problems, quickly.

Many people think ( that included me, too) that Test Experts do lot more tests and have more fantastic test ideas and that's why they find many important problems, quickly.

I was hoping that there is more information hiding than what is visible about test experts and spying helped to uncover those hidden secrets.

This post is all about the secrets that I found spying test experts and the secrets of how they find many bugs?

You can trust me about the information to an extent that I and many other testers have been practicing the secrets that has resulted to our success of finding many important problems, quickly.


They just do one test to find many bugs!

Now, I am sure you don't want to believe me because my finding says that they do just one test to find many bugs. I am sure you have seen testers finding one bug per tests that find a bug and that makes you not believe this information but... you must understand that the secrets are always unbelievable.

You can feel safe after knowing a fact that I too didn't trust the secret of one test finding many bugs but my spying mission was to find out more granular details about it.

Looking more carefully into it, you might discover that it is not a test that finds a bug but it is a human that finds a bug and a test plays a role in helping the human find it. So there is something with humans that is helping to find bugs and not completely with the tests.

When I collected this much of information over months of spying, I was invited to be a part of the round table conference of experts as I had posed myself as a test expert from India (I knew I wasn't a test expert and even today I am too conscious that I am not one such) but you know... I am a spy and had to pose that way.

Over the round table conference, test experts started discussion about ORACLES* and I didn't know their terminology and just took notes. That evening, when I returned to my hotel, I was trying to connect ORACLES and the spying mission I had in hand.

* Oracles - principle or mechanism by which we (humans) identify problems

I was perhaps lucky that James Bach and Michael Bolton were staying in the same hotel that I was put up and we caught up at the dinner table.

I had an idea that I hope it worked. I was hoping that James Bach is a kind of person who would spill off the secrets in a context to prove me wrong and help me learn the secrets if I talked nonsense about ORACLES, because his passion to the craft of software testing is something that I am trying to match. Michael Bolton, the Birbal of North American software testing was a person whom I had to handle very carefully as he is more likely to identify me as a spy. At the end, I assumed that I was able to outsmart Michael in not letting him know that I am spy.

Going back to the hotel room, I started writing my discoveries about the secrets of test experts and the granular details of how they find important problems,quickly.

Before I publish a report to end the spying mission, I was sure that I had to practice the details I collected. I did practice and was excited that those ideas really worked and I found more bugs by executing one test.

Why many testers might not be able to practice the secrets that I am unlocking here is because they have a fixed expected result that is derived from a specification document that the tester thinks is The Holy Bible for the project.

Specification Document - is one single oracle and hence you might find one bug per test you execute.

The important thing is experts define a bug as "anything that threatens the value of the product" (-- James Bach)

Here are a list of Oracles that experts use with every test they do:

Consistency with the Image: When I execute a test, I try comparing and contrasting it with the image that my company or stake holders have been projecting about the product or the company.

For instance, if my company has an image in the market as people who produce software whose performance is good, and I execute a test whose documented expected result is "This functionality should work", I ask question, "Well it works but didn't it take too long to perform the operation?" and then I find a bug and say, "Although this operation goes through the time it took to complete the operation exceeds beyond the image the market has about us"

Consistency with Similar Product(s): When I execute a test the functionality might work as expected but I ask a question, "Well it works but doesn't it seem to perform in a way that users who are used to similar products might expect it to happen?" and then I find an important problem, quickly.

Consistency within the Product: When I execute a test that appears to have passed, I ask a question, "Well it works but doesn't it work in a different way than other areas of the same product?" and then I find an important problem, quickly.

Consistency with Claims: Marketing personnel and my manager are making certain claims about the product. When I execute a test and the test appears to have passed, I ask a question, "Well it works but does the way it works go against the claim that they make?" and then I find a bug.

Consistency with Statutes: When I execute a test that appears to have passed, I ask a question, "Well it works but does the way it works go against a law that an organization or a certifying authority might have?" and then I find a bug.

Consistency with User's Expectation: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in a way that the users whom I am aware of might not want it to happen?" and then I find a bug.

Consistency with History: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in the way the previous releases of this product work?" and then I find a bug.

While writing this post, I got an e-mail from James and Michael that reads...

Dear Pradeep,

We knew that you were spying us for a long time. We also saw the way in which you took notes, asked questions and drew inferences from what you could capture and that impressed us to help you learn more about testing because those are some important skills that a tester needs to have, to add good value for the cost customers pay for. We consider to hire you and we recommend you to quit spying profession and take up testing where you might get more laurels for your skills. Here is a link to know more about more secrets : www.satisfice.com/rst.pdf

Good luck!

-- James Bach & Michael Bolton

I was excited and I did quit spying, to take up testing as a full time activity. I thought I shouldn't post this in my blog because I didn't want those secrets to be given away for free and saved a draft of this post. I am afraid if Blogger has a bug that publishes drafts that are not intended to be published.


-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton


Update : Michael Bolton, blessed me with a post on his blog for this article of mine. I was jumping in joy for at least half an hour reading that and I am pleased to share it with you: http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html

21 comments:

Anonymous said...

As always you make a good reading time, nice article, and I feel the same way sometimes, trying to figure out what makes them a good testers, to try to teach my team to become ones.

See you.

Pradeep Soundararajan said...

@Jose,

I am very happy to receive your feedback as the first ones and I hope you would continue to do so.

It took me 4 hours to craft this "draft" and I feel happy that it has made a good time to you and your team, maybe.

Keshav Ram Narla said...

Looks great. But i will have to go though this again! ;-)

Thanks for your time.

- I'm curious how you can manage posts like this in your daily schedule.

Anonymous said...

Interesting Post Pradeep! I loved the way you compiled and presented it. I am not sure about the Experts, but some good testers might be already following some of those oracles. Anyway, thanks for spying! :D

Incidentally I have written something on a similar flavor in my blog recently. Here is the link (Top 5 Secrets to Bug Hunting Success!) in case someone is interested.

--Debasis

Pradeep Soundararajan said...

@Keshav,

I'm curious how you can manage posts like this in your daily schedule.

When there is passion, a day looks: like 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour + 1 hour!

Meeta Prakash said...

hey pradeep ....... good one ......but dont be scared of thiefs ......how long can they sustain with no substance and foundation ?

Pradeep Soundararajan said...

@ Meeta,

Thanks a lot! It is a honor that people like you come to spend time here.

Being scared of a thief could also be a technique to give him a surprise attack :)

Anonymous said...

Michaels blog post on this article is a great achievement as a tester from India and you will make all of India proud.

Pradeep Soundararajan said...

@Anonymous,

Thanks for your compliments. I wish to make India and the testing community proud.

I see great testers as philosophers, who can influence the world and Michael is one such. As I put on Michael's blog my thinking is influenced by a lot of people and the list is huge.

I also wish to thank Sir Thomas Alva Edison who changed my life. Long back, I was considered as a dumb person and I did not believe it. Today hardly people say that but I do believe, I am dumb, because there is so much to learn and I am left behind.

Thank you Sir Edison and thank you all so much!

Anonymous said...

Hi Pradeep,
This Post is good and the list of Oracles that experts use are also good, but i would like to share my real time experience, where we have dynmaic projects working for aggressive clients expecting us to give releases every month and going on adding change requests till the last moment, they just expect that all their user funcionalities are present or not that's it. Yes sometimes they care only about the performance in the list of oracles, but they won't tell it is an important bug if we find out by considering other types of oracles. In Reality they just won't encourage us to do or tell anything apart from the Functionality and Performance. After the Release If User does not come up with any defect in the application they tell our Testing Team is Good,In the next Release if User comes accross even 1 defect also immediately they complain our Testing Team is not good and they add lots of Action Items for us.

Pradeep Soundararajan said...

@Anonymous,

I am not sure if your customer understands what testing is and I am also not sure if your team knows to educate customers to not manipulate the oracles you have.

I have worked with such customers BUT I haven't allowed them to decide what I should do.

However, I am not sure whom you mean by client and I encourage you to write to me over e-mail if you really want insights about the issue that you are facing. (provided you think I can give you insights)

I have a notion that you might not have understood this post and I always know I can be wrong.

Shrini Kulkarni said...

Michael Bolton once mentioned about HICCUPPS consistency heuristics that is similar to what you are mentioning about oracles.

As you have rightly mentioned, most of testers today use only one
(some time insist that they use only that) oracle to find the problem - specification.

Real bugs in software are too tough to be identified by a very routine and mundane oracle like specification. If it were so, software testing would have become so easy. Just write test cases *confirming* specification and execute them - find all bugs.

Unfortunately - many today's software world think that is all is testing. Fortunately for few of us, we know that testing much more than that.

It is time for all testers to constantly look for, design and work with different kinds of oracles.

Shrini

Pradeep Soundararajan said...

@Shrini,

Michael Bolton once mentioned about HICCUPPS consistency heuristics that is similar to what you are mentioning about oracles.

I didn't do any big job. I just copied the titles of Oracles from Rapid Software Testing slides and have linked to them, too.

Real bugs in software are too tough to be identified by a very routine and mundane oracle like specification. If it were so, software testing would have become so easy. Just write test cases *confirming* specification and execute them - find all bugs.

What do you mean by real bugs?

Anonymous said...

Hi Pradeep,

This is a good article...
But really when we speak of 'Product Consistency' w.r.t testing any product that the Tester:
- Should be conversent with the domain
- Under stand the product/how it works
- Understand the business behind building the product.

But all the above depends on the ability of the tester to understand or the amount of KT given by the company regarding its products.

Hence if a bug is found in 'UAT' then the company points fingers to the testing team for having missed this functionality.

Hence does'nt it mean that the tester should fisrt UNDERSTAND the
domain/functionality and then do testing.

Also frankly by experience, its not correct that a tester wll be knowing the entire product/functionality.

So this being the plight why do testers take the blunt, is it because they are the people who certify the product?


Appreciate you valubale comments in this regard.

Pradeep Soundararajan said...

@Anonymous,

I am hiring you to test a missile that I have developed and you dont know anything about missiles.

When you press the Fire Missile button, nothing happens. As a tester would you report that as a bug?

The way people around you are showing something as "testing" is actually not testing in my world.

Excellent testing doesn't necessarily mean a bug would not be found during UAT. If your definitions of testing is wrong, you go by your definitions which lead you to the way it is intended to lead.

However, domain knowledge is important but not the most important thing to test well. I speculate that the people around you might not know what testing skills actually means.

Anonymous said...

Yes, as a tester surelly this will be reported as a bug.

Then what is 'Testing' actually mean in ur world?
Can you pls share this as it will be useful for everyone to read.

But the definitions can vary w.r.t to the context of what and how you want to test, but certian things mean what say.

Domain knowledge is very important as that will add value to the testing [in finding bug] and the product.

According to you what is testing skills then?

Pradeep Soundararajan said...

@Anonymous,

Thanks for coming back. Your comment gives me a clear idea that you are a new reader of this blog. All the answers that you might be curious to know through your questions are available in my blog.

I would suggest that you either spend time reading other posts or continue reading the posts for a while.

Shrini Kulkarni said...

Real bugs ---

Ones that make you slog (think) to find them.... they are too smart to be identified by scripted - confirmation oriented test cases.

When you use oracles like spec (alone) you can only notice bugs that are related to the spec that too once or twice not all the time.

My view is expand you collection of oracles and increase the chance of finding the bug every time

Shrini

Pradeep Soundararajan said...

@Shrini,

I politely disagree to your notion of "true bugs"!

Even if I dont think and hit the enter key while testing a software and find a crash, it is a true bug in my opinion.

Maybe you want to re-phrase it to help me understand what exactly you want to mean.

Mubbashir said...

Great post again!!
as Debasis
pointed out some might be (consciously or unconsciously) )following these but from now onwards i will make a conscious effort to follow to "find many bugs from one test"

Regards
Mubbashir.

Unknown said...

Its a kind of eye-opener for testers who thinks testing around the requirement is the primary task and following other oracles is optional If time permits.