"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Monday, January 10, 2011

Evolving styles & small "e" exploratory testing

When I first heard about the small "a" agile and the big "A" agile, I was intrigued. I was wondering if it really did exist. Later when I interacted with those who were agile and those who claimed to be agile but were not, I started believing in it.

Similarly, I am going to be talking about the small "e" exploratory testing style.

Every tester I have interacted, tested in pairs, watched them test... have their own style to testing while being exploratory. My own style has changed over the years and its evolving. To detail it out, there are several dimensions.

First dimension

  • I Did exploratory testing without knowing it was called that way. To...
  • Did learn the term exploratory testing but didn't know if that's what I was doing. To...
  • Did exploratory testing based on my own ideas of how it should be done. To...
  • Did learn that others have their own style to exploratory testing. To...
  • Did practice other testers style of exploratory testing combined with my own style. Plus...
  • Learnt that to become a decent exploratory tester, I need to develop various kinds of skills. Plus...
  • Started to ask for mission, charter, objectives before exploratory testing. Plus...
  • Learned about SBTM and practiced exploratory testing with SBTM (and failed at my first trials). Plus...
  • Learned to modify SBTM to my style and my context needs (current state). Plus...
  • < To be updated >
  • ...
Here is the second one.
  • Was clicking on every visible GUI stuff to see if a bug dances out. To...
  • Thought I was doing good exploratory testing if I was finding a lot of crashes. To...
  • Thought good exploratory testing should always result in finding a lot of bugs. To...
  • Learnt that bug investigation is purely an exploratory process. To...
  • Learnt that test designing skill is important and learnt & practiced it. Plus... (note the plus?)
  • Learnt being conscious of the test coverage is important. Plus...
  • Learnt that making notes of all my observations and inferences is important. Plus...
  • Learnt that briefing and debriefing is not just like any other discussion about testing. Plus...
  • Practicing to debrief. Plus
  • < To be updated > 
After second, usually its the third dimension
  • Thought of exploratory testing as a technique. To...
  • Learnt exploratory testing is an approach. Plus...
  • Practiced applying several techniques in exploratory approach. Plus..
  • < To be updated >
The fourth
  • Unable to explain what I did after a few hours of testing. To...
  • Being able to verbally explain what I did. To..
  • Being able to explain and document what I did and why I did that. To...
  • Being able to do that much better over practice. Plus..
  • < To be updated >
Then comes the fifth
  • Being able to do all that and trying to keep it secret because I wanted to be a hero. To...
  • Realizing that I can better only if I share with others. Plus
  • Started to coach testers. Plus
  • Started to learn from them. Plus
  • I recognized that if I had kept it as a secret, I would have been zero and not hero. Plus Plus,
  • < To be updated >
Semi finally, the sixth
  • Aping what others are saying. Plus
  • Trying out my own ideas. To
  • To forming my own style of doing it. Plus
  • Trying out different attack (not the attack you know) as a starting point to my testing. Plus
  • < To be updated >
  • Thinking of all exploratory testing sessions as freestyle. To...
  • Learnt that there are loosely scripted & checklists types. Plus...
  • Practicing loosely scripted, freestyle, checklists types. Plus
  • Learnt that a good exploratory tester cannot be determined in the first few hours of testing but what comes after that. Plus
  • < to be updated >

If you were wondering why I moved from one style to the other, I want you to be informed that it was for being more value to my customers and not because someone said so. If you are doing anything because someone said so, without knowing if it suits your context or what value it adds, you are most likely doing the big "E" exploratory testing. Also, if you don't know what many serious testers who are practicing to be good in exploratory testing are talking / practicing, you still are doing the big "E" exploratory Testing. If your style of exploratory testing hasn't evolved over the years, you are most likely to be doing the big "E" exploratory testing. Nothing wrong in being the big "E" Elephant. Everybody started there. Everybody were stuck there. Not everybody remained there for a long time.

Also, you may ask if the points listed above indicate anything about the style. In case you did ask, I want to thank you for that because I plan to write about it here. I mean, here --> As we learn and unlearn, the way we test, changes. This influences the style. 

For example, imagine me thinking of all good exploratory testing to be yielding tons of bugs, I'd grow to be a manager who'd question someone why they didn't find a lot of bugs over the last one hour of exploratory testing and hence making myself stupid.

If you were to take another example: Learnt being conscious of the test coverage is important : Everybody in their first few years of testing, at some point, lose the sense of test coverage they are achieving. They get carried away by bugs and as a matter of fact, the client is happy too because they are seeing important bugs being found. Currently my central theme of exploratory testing is test coverage. I ask myself, "OK, I found these bugs and they are great but what is happening to my test coverage". The client may not know to ask for it but it is my duty to help them with that information.

This one, Yo! Being able to explain and document what I did and why I did that. -  I personally know many testers who seem to do good hands on exploratory testing but being unable to narrate a story of what they did, makes them look bad. The senior management appears to just look at the bugs they report and hence the good exploratory tester looks like a "not so good" exploratory tester. Today, I can tell an engaging story of why I did what I did and what value it adds to them and also why I should be given more freedom and hence more responsibility.

Try this! Practicing loosely scripted, freestyle, checklists types. Plus

Based on how accountable I need to be, I choose which type of exploratory testing is needed. That only comes with practice.

Recently, I was asked to explain my test coverage & strategy. Why did that arise in between the consulting assignment? When I started my first round of testing, I was finding bugs as I learnt the application, then I stopped finding bugs because I was doing some confirmatory tests and during that time, the client was wondering, "Is that all about Pradeep?". A very good question to have asked. So, I presented them with my strategy and coverage plan. Here is snippet from that. (Taking out all confidential information). Level 1.

- Toolbar options
- Mouse hover / clicks / right click / left click
- Keyboard shortcuts
- URL / Path editing
- Menu options coverage
- Claims based testing
- Questioning the purpose of existence, position, alignment and meaningfulness of all UI
- Consistency within the application
- Consistency with Windows standards
- Confirmatory testing
- Process monitoring & control
- Security vulnerability
- Consistency of text, font, sizes, options
- Links / Broken links / paths
- Reliability of user login / logout
- Violating policies - Firewall, System, Third party software
- Hidden details & options
- Scenario based tests -
- Boundary tests
- Hardware based scenarios - Interrupt
- Timing based coverage
- Error message coverage
- Tools / Utilities
- Independent components
- Extended usage
- Multiple connect / disconnect
- Reboot / Sleep  / Hibernate
- Close abrupt / Kill
- User coverage
- Usability standards comparison
- Functional checking
- Supportability testing
- Accessibility
- Navigating between zones / desktops / sessions
- Platforms
- Devices coverage - Bluetooth, Printer, Memory, Ext HDD
- Authentication / Authorization
- Splash screens
- Obvious / non Obvious things for a user
- Brand image versus product behavior
- Comparable products / benchmarking
-  Testability tour
-  Behavior patterns, user expectations and consistency

At Level 2, I had a checklist of what tests I did.

Here is an example: http://support.microsoft.com/kb/301583 : I ran through each one of them. I could tell what shortcuts cause what problems and an investigation about the problems. They were highly convinced why they were paying me as much as they did. I just stood up to the scrutiny.

At last, Trying out different attack (not the attack you know) as a starting point to my testing. Plus

Most testers use Functionality / Security as their first few possible ways to start attacking (start testing) a product. I was practicing to try out different things trying to determine how useful it is for short and long term projects. Testability attack has yielded great results for me than any other form of attack. Now, I might have learnt a lesson that the world already knows but what matters is, I did learn it. Hurrah!

If you were to determine how skilled an exploratory tester is, its not the first few hours, its what comes after the first few hours. That's what determines the the big "E", exploratory testing style and the small "e" exploratory testing style.

Also, our testing industry is so damn confused that they might end up asking you about the big E and small e exploratory testing in interviews by those who might happen to read this blog post half way and run to a meeting. Tell them that Pradeep used the small e and big E to explain styles and not to define or redefine anything about exploratory testing. Also, tell them that they didn't read this post completely. 


Risko said...

Thank you for the read. A really enjoyable post. I am currently trying to move from the big E closer to the small e and I think this post made this transition much more clearer for me : )

Anonymous said...

Now about Testing and testing :-)
-Dhanasekar S

Selim Mia said...

Good read as always :)
This small "e" exploratory testing style would build up tester's credibility and add more accountability of each exploratory testing session.

Thanks for sharing your formulated thoughts with us *thumbs-up*

Fake software Tester said...

And when you write your book on software testing, I hope you add the above items to it. Great checklist elements!!!

Krishna Gurumurthy Punekar said...

Thanks a ton for sharing your ideas on Exploratory testing.

I agree with you 100% on skills for exploratory testing comes only by practice.

Before I start with et, can you explain what do you mean by the below statements.

1. You have mentioned that "to become a decent exploratory tester, I need to develop various kinds of skills. Plus..."

What are those various kind of skills to be developed, please elaborate. Should one understand those skills and then do exploratory testing, let me know.

2. Started to ask for mission, charter, objectives before exploratory testing.

Is this question for yourself? Otherwise, who has to take call on what the mission should be?

How does charter, objectives influence the way I do et, if possible can you explain with an example?

3. Learned about SBTM and practiced exploratory testing with SBTM (and failed at my first trials). Plus..

How long did you do practice this and what do you suggest for others (say for a person who is in level 1 according to your checklist above)

4. Learnt that test designing skill is important and learnt & practiced it. Plus... (note the plus?)

How did you practice this, can you share?

5. Learnt that briefing and debriefing is not just like any other discussion about testing. Plus...

What do you mean by debriefing while doing et, please explain.