"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Wednesday, March 21, 2007

Mixing Scripted and Exploratory Testing

Over the past few days, I have been talking and listening to a lot of testers, leads and managers in Bangalore IT organizations through my Mirchi Test Masala talk and it's a great experience.

I solved interesting problems that boosts my confidence of solving testing problems that Indian IT companies face and I have already started to receive great feedback from testers and managers who attended my talk. I usually don't believe on the feedback forms but I do believe when people talk the change happening in their workplace and that is always a true feedback.

One such problem that I solved for an organization is this:

Manager: I usually ask my testers to focus on finding issues that are important to the release and I see that not everyone in my team finds those bugs that I/management/customer is interested to see with that particular release we make.

Tester: Well, we do follow what he says but he is not being convinced about the work we do. We are finding problems but the problems we find are not of the stake holders interest sometimes.

Manager: There was once a design and code change to optimize performance and we did not have the test cases ready and I had to ask them to do it in an exploratory way but I could see that it did not work well for any of us.

Tester: Thanks that our manager communicated that he is interested to listen more about performance related issues on a specific release and we did exploratory testing but we are not sure why we couldn't find bugs.

Pradeep: Ah! Interesting problem!

This is what I did:

  1. First I launched an explanation of my understanding of the problem and asked them, if I understood the problem.
  2. On getting a go from both sides, I started to question them individually.
  3. The questioning skill plays a vital role, in seeking more information, analyzing the problem and get hold of contexts in which they are speaking.
  4. I asked the manager and testers a couple of common and uncommon questions and made them to listen to each other.
  5. A specific question that I asked, "What is your definition of performance and exploratory testing?" revealed the solution to the problem for everyone in the room.
  6. Interestingly, the manager had a different definition of Exploratory testing from each of the tester. The testers too had their own definitions of Exploratory testing and it differed from each other.
  7. I insisted the testers to hereafter ask for what the manager meant by a word/terminology when he assigns them to do a specific task.
  8. After that I got them off to a discussion based on a topic over which the manger thought his testers fumbled upon and did a bad job. Interestingly, both ends were now convinced that they could find those important problems quickly since they questioned each other.
Now, there is another problem that testers in all organizations I have been so far and I am going to be for my one day testing exercises workshop will ask my help to solve:

How to mix scripted and exploratory testing?

I understand the fact that Indian IT companies do not want try exploratory testing as a full fledged activity for their projects unless they see results for themselves and they still want to try it out to see if it adds value to their testing.

In one of the projects I worked, I mixed scripted and exploratory testing and the results that the organization saw was amazing. Ah! it worked for me but I can't guarantee that it would work for you but I can guarantee you that it's worth a try.

Before you know how to mix scripted testing and exploratory testing, I suggest you to go through Jon Bach's video on Exploratory Testing and RST appendices . Now, if you crave for more you might want to have a look at Elizabeth Hendrickson's Test heuristic cheat sheet and form something like that to suit your project needs. Of course don't forget to run my screen saver which is now installed and used by all testers in an organization and I am happy to hear that it has been helping them.

Now, it's time to know how to mix it without adding too much cost to the organization and yet find those important problems, quickly.

For every scripted test case you execute, run a test that is not documented in your test case suite, which is fast enough to run and yet not add too much of time. You might want to consider using heuristics for getting on the fly ideas to test the product that you have been asked to.

It's left to you and your skills, to decide the ratio of mixture of Scripted test and Exploratory test and also don't get surprised at the results as you keep practicing this and learn to do it much better.

Rakesh VK, a Senior Tester in DELL is well known within his company for being the tester who finds most number of important problems quickly. While delivering my talk at DELL, I had a chance to talk with him over the secret of being the tester who finds problems that matters the most.

He simply had to say this: "I mix scripted and exploratory testing and I always have been a tester who finds most number of important problems and I enjoy remaining in this position. Also, this has led my manager to get convinced that ET adds great value and he now is happy of other team members doing ET"

Wow! So who is the next Indian tester to become one like Rakesh?

-- Pradeep Soundararajan - pradeep.srajan@gmail.com - +91-98451-76817

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton


Ukkuru said...

You are true! Most of the managers Indian or US does not understand the need for exploratory testing! This method is very good in finding defects with minimum cost! It is not really worth to spend too much of time writing test cases and reviewing them . That valuable time should be utilized for testing so that customers will not have an oppurtunity to find them

Skiller said...

This is a very interesting post. Motivated me to exploratory testing and use cheat-sheet and find new bugs.

I shall keep you posted on the new bugs I find with this mixing and your Masala formula is cool :)

PP said...

Hi Pradeep,

I guess this Test Masala talk is increasing your and Blore testing community’s arsenal exponentially. You must be getting new types of scenarios,problems,attitudes…Wow..May be having time of your life. And have no doubt we Indian tester community are getting to know the more correct way of carrying out the testing.

I enjoy the way you critically analyze every testing problem or attitude problem we have in India (sometimes globally).

The method you suggested mixing scripted testing with Exploratory testing was something I always used .In later rounds of testing, exploratory testing was the method that yielded more bugs than scripted but as we were supposed to finish all test cases for round to finish…..It was the only choice we were left with …that is.. mixing Scripted with Exploratory testing. Now when you mention I realize it is something I should always stick to. Albeit Pradeep,I am not happy with the way I do exploratory and that’s why started reading James and your blogs.It was initially surprising to know that there is a scientific way of doing it.

James always stresses on trying to constantly evolve our process of thinking.When he says its better to take every testing exercise keeping context in mind ..there is hidden truth that understanding the context will make our thinking process more clear and help us to test better.


Pradeep Soundararajan said...


I delayed in replying to you!

Albeit Pradeep,I am not happy with the way I do exploratory and that’s why started reading James and your blogs

I am not happy too and that's why I try finding more better ways of doing it despite the wonderful results I have been able to obtain from ET. I explore more ways and skills of doing ET and that's what you should be doing too, in my opinion, to get happiness of doing ET better each time and yet be unhappy, to improve it further next time.

PP, I would love to receive an e-mail from you or have I known you by some other name than PP?