"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Monday, January 08, 2007

Testing exercises, found everywhere

Hi Reader,

After getting Satisficed, I find myself craving for more testing exercises. I have started to work on my own exercises for coaching my full time student Rajesh.

The exercises are usually ad-hoc.When I say ad-hoc, I mean we both go out for a walk and I see something, get an idea and throw an exercise to him. I attach a testing context to it and I set traps. A tester is supposed to clear the traps and accomplish the mission.

Such exercises are helping me to think of ideas, set traps, clear traps and pair testing. Michael Bolton, once gave me an exercise, 2+2=4 and it is that exercise, which made me realize exercises are everywhere.

Here it goes, "A bullet and car glass exercise" that Rajesh and I were playing yesterday:

A man fires a bullet from his gun, aiming at a car window glass. The glass remains undamaged.

List the possibilities for the glass remaining undamaged.

(you might want to work on the above exercise before you have a look at our answer)
  • Glass is bullet proof
  • It was a toy gun with a plastic/rubber bullet.
  • Aim was not proper.
  • Misfiring.
  • Car moved.
  • The object did look like a glass but it wasn't.
  • Someone inside the car, opened the door/window.
  • The man shot at the reflection of the car/glass window.
  • It was a mirage.
  • The barrel was bent in an angle.
  • Someone/object/bird/animal, came in between.
  • Car was protected by an invisible shield ( As shown in the movie "Independance Day")
  • The man shot from a distance more than the range of the gun.
  • The man shot the bullet from underwater, the bullet lost its potential or momentum to break the glass.
  • The car was in a paper advertisement and the person reading the paper changed the page by the time the man fired a bullet.
  • In some Indian movies, the hero catches a fast moving bullet with his bare hands. There is a possibility that an Indian hero caught the bullet before it hit the glass. :)
  • The bullet was deviated due a strong wind/storm.
  • The time the man was pulling the trigger, someone shot him.
  • The barrel was blocked.
  • The glass we are talking is not the same car window glass.
  • The man always missed his aim by a couple of metres.
  • The car was inside a showroom protected by transperent bullet proof doors/windows.
  • <>

It appears to me that such an exercise, helps a tester in coming out with test ideas, brainstorming the possible situations of a bug found by the customer, reproducing the bugs that are found by non-test teams/clients/end-user.

I see many testers struggling for ideas. Working through such exercises can improve spontaneous idea generation, which helps when practiced and put in testing projects.

Here is another one:

The image you see above, is a screen shot taken from PuTTY. I had enabled logging the session and the image you see above pops up, each time I try to login.

You might observe the options "Yes" , "No" and "Cancel". If you take a careful look you might also observe a "Close" button "X" on the top right corner. Now, work through this:

  • What is the difference between "Cancel" and "Close"?
  • How would you test "Cancel" and "Close" functions?
  • If "Cancel" and "Close" produce the same result to end user, can one of them be removed?
  • Why would a user prefer to have both?
  • If you were to design this window; what would be your design considerations?
  • What evidence would you give or look for to differentiate "Cancel" and "Close" as two different operations?
  • On finding an evidence that "Cancel" and "Close" are two different operations, would you log a bug for a situation where you are testing a different application that has "Cancel" and "Close", as same operations?
  • What are your test ideas and oracles to test this? ( Oracle: is a principle or mechanism used to recognize a problem)
  • List all possible applications that you have toured and tested, which has a "Cancel" and "Close" and also try remembering whether all of them behaved in the same way.

Asking a lot of questions like these helps you develop your questioning, clears assumptions, find bugs and see the smallest things in front of you as the most challenging and learning opportunities.

You have exercises everywhere. If you need to see them, you need to have a passion towards the craft, a passion to better yourselves as a tester and "(smart)testers way of thinking" and willingness to help other testers.


"A good testing exercise, reveals a hole in your education" - James Bach

(I vaguely remember James saying this to me, if I worded it wrong, James would correct me)

Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

12 comments:

Anonymous said...

---Hmm, difference between Close and Cancel:
None

---How would you test "Cancel" and "Close" functions?

First, make sure that when you answer Yes or No to the question, putty.log changes (size of file is the first thing I think of and an easy way to see something happens - you could look for content too, but it is more difficult to see if something has changed).

Test Cancel and Close begins here
1. Look at putty.log (size of file).
2. Try to open session, hit Cancel at the dialog.
3. Make sure size of file remains the same.
4. Repeat the steps above but hit X instead of Cancel.

This is the first test I have in mind, I could find some other tests but I think this would be enough.


---If "Cancel" and "Close" produce the same result to end user, can one of them be removed?

---Why would a user prefer to have both?

Of course, one of them CAN be removed.

The question is if removing one of them would make the users happy or not.
Removing Cancel would confuse some users.
Removing X would make me think the person who made the Window is not very good (I see the X button as "more powerfull" when I want to close a window - it sould work even if the Cancel button doesn't work - this is only a subjective perception).

---If you were to design this window; what would be your design considerations?

1. Who uses PuTTY?
2. The average users of PuTTY understand the window?
3. Is the window clear to any person who understands Eglish and has some knowledge of how computer programs work?
4. Are the options clear?
5. Are the options enough? (I mean, should I implement some other behavior besides replace, append and turn loggin off?)
6. How do other windows with the same options work?
The order I listed the things above is not the importance I give them.
I could think of more, but I am not a designer and for me the things above cover enough.

---What evidence would you give or look for to differentiate "Cancel" and "Close" as two different operations?

1. I would look for difference in putty.log.
2. I would look for difference in settings: clicking Cancel disables logging for this session, clicking Close could disable logging from now on...
3. Clicking Cancel disables logging for the session and starts putty, while clicking Close just shuts down putty.
4. Clicking Cancel disables logging for the session, while clicking Close does something else - anything else - difficult to catch (like turning the light on in the room I am :), changing the world weather and so on :) ).

---On finding an evidence that "Cancel" and "Close" are two different operations, would you log a bug for a situation where you are testing a different application that has "Cancel" and "Close", as same operations?

I would file a bug, with severity and priority depending on the behavior of Cancel and Close.
I would not consider a bug if clicking Close closes putty, but I am sure some users will be confused, so i will file a bug.
And of course, if clicking Cancel turns logging off for the session while clicking Close breaks something (or viceversa) is a bug.

---What are your test ideas and oracles to test this?

Test ideas:

Exercise all buttons and look for behavior - putty.log file.
Exercise behavior when using keyboard to navigate through the buttons.
Exercise behavior for every option when putty.log is deleted after the window opens.
Exercise behavior when user tries to open another putty session for the same machine before clicking any option on the window, then chosing different behavior for each - what is the outcome?
Interaction with other dialog windows (do other dialog windows interfere with this one - from within putty or from another application??? - not a very likely situation, but if it happens, it is a very serious problem :) ).
Trying to close the window with Alt+F4.
Moving through windows with Alt+Tab.
Move window out of the screen, change resolution, try to reach the window again (by any means, using mouse, alt+tab, etc.).
That is what I thought of in 10 minutes, I am sure I can come up with other ideas :).

Oracles:

The first oracle I had in mind is properties window in Windows (right click on desktop, then click Properties). This has Cancel and Close, so for testing just Cancel and Close this would do it.
Second, I will look for window which has Yes, No, Cancel and Close buttons on a popular application and see how it behaves.
I have some oracles made in the company I work - windows with Cancel and Close buttons, windows with Yes , No, Cancel and Close buttons, seem to be harder to find :).
And of course, one of the oracles would be my "expected result", meaning what i expect to happen when i use the window.

---List all possible applications that you have toured and tested, which has a "Cancel" and "Close" and also try remembering whether all of them behaved in the same way.

Many, first coming into mind Properties window in Windows (right click on desktop, click properties).
Many dialog windows with Cancel and Close.
Many dialog windows with yes, no, and close.
Not so many dialog widows with Yes, no, Cancel and Close :).
I will not make the list right now, but I think it would have at least 30 examples, and as far as I remember all behaved the same way regarding Cancel and Close at least.

I did not list all windows because I just answered this as a fun exercise, not spending more than one hour on this. If I would have to do this thoroughly it would take from 2 to 6 hours, 30 minutes just for searching for windows with yes, no, cancel and close buttons, insisting more on test ideas, on what I would have in mind designing a window, asking other people on design ideas and on expected behavior (people with not too much knowledge of computers) and so on.

Regards,
Stuiber Victor

Shrini Kulkarni said...

Pradeep and Victor -- nice to see a great questioning going on ...
Please keep it up ...

Shrini

Anonymous said...

http://truestesting.blogspot.com says:
1) Cancel and Close
As Told in the message Box Cancel will disable you logging i.e. might be you might have to reconnect yourself. Whereas the Close button will leave you unchanged with what you are doing at the moment.
2) How would you test "Cancel" and "Close" functions?
First check the Log File. Cancel might be deleting the log file and close will let be remained as it is i.e. before and after the event..
2) If "Cancel" and "Close" produce the same result to end user, can one of them be removed?
Here Cancel and Close will not produce the same result.
3) Why would a user prefer to have both?
In all the Pages we see normally we have these options. because if some one is happy with key board with Tab order he can reach out cancel button or if some one is happy with mouse he can go and click on the close button and to help both types of users we usually have these type of buttons
5) If you were to design this window; what would be your design considerations?
No Idea.
6) What evidence would you give or look for to differentiate "Cancel" and "Close" as two different operations?
As Told in the message Box Cancel will disable you logging i.e. might be you might have to reconnect yourself. Whereas the Close button will leave you unchanged with what you are doing at the moment.
7) On finding an evidence that "Cancel" and "Close" are two different operations, would you log a bug for a situation where you are testing a different application that has "Cancel" and "Close", as same operations
Obsolutely depends on the requirement.I will defintely know in advance the functionality that is supposed to happen fo r both the Cancel and Close Buttons.
8) List all possible applications that you have toured and tested, which has a "Cancel" and "Close" and also try remembering whether all of them behaved in the same way.
In all of the current pages I Test Cancel will be get to the Unsaved Data (if any ) or to the default values which each individual field is expected. Close will close the Page.
Regards,
MrSharma56@gmail.com

Anonymous said...

Nobody else?
Please, it is a fun exercise, and I really want to see what other can come up with so I can improve myself.

Regards,
Stuiber Victor

Anonymous said...

Good work...

Krishna

Anonymous said...

Pradeep,

Nice work.
I recently did some cleanup in my cabinets and stumbled into a CD with the game Myst. The game reminded me of the friend who gave it to me. I put the CD aside. Later I was reading some post on the context-driven yahoo group and I though Exploration ...Myst.

I came up with the idea to use these kind of games to practice my note taking. I am training my self in SBTM. I believe this kind of exercise would come in handy. I hope to complete a test session for Myst this month. When I get that far I'll post it on my blog.

Erwin.

Anonymous said...

Pradeep,

as far as "A bullet and car glass exercise" goes following questions.

How do you know that the man fired the bullet, and how do you know that the glass is undamaged?

Did you see it?
Did you hear it?
Did someone just tell you?
did you read about it in the newspaper?
Did you read a police report?

I always try to ask myself: How do I know what I know.

Erwin.

Pradeep Soundararajan said...

I always try to ask myself: How do I know what I know.

That's a good question to ask ourselves.

It depends on what you are trying to know . For example, if I am testing a product I can infer/conjecture about the product behavior to the test I do BUT I know what I did. Self awareness can lead you to say "this is what I did" and being a good enough tester you might want to say, "I conjecture this is what I know about the product".

Anonymous said...

Speaking of which, Pradeep, it seems to me that you can develop a game out of this. A training game with appropriate number of scenarios that the gamers can both have fun and learn. That would be a wonderful educational tool.

yan

Anonymous said...

Maybe there is not supposed to be a difference. In that case the problem is that the worng button label has been applied and the button should have been labeled "Close" instead of "Cancel"

Shrini Kulkarni said...

>>>I conjecture this is what I know about the product"

Let us for a moment mull over what Erwin Van Said -- "How do you know what you know? - epistemology". How do you what you know about the product? Is it what you have seen? heard? felt? imagined? deduced? The mechanism or means with the help of which you *know* can make a big difference in "what" and "how" much you know.

Knowledge about something that we are focusing on -- is multidimensional - human beings use all sensory and extra sensory (Brain?) organs to gather the information and Brain processes them and tags them as "knowledge".

So question each source of information, question each method of aggregation and disaggregation of information - may be you will get know more about what do you know...

You must have heard about inattentional blindness - related to "see" aspect of our observations. When you say you are seeing the computer monitor - you are actually seeing some attribute(s) of the screen. In most of the times, you will be become blinded for certain glaring items in front of your eyes if you are not paying attention to them.

Where all these takes us to?

Shrini

Nisha said...

In this specific dialog box i agree with this comment:

Anonymous said...

1) Cancel and Close
As Told in the message Box Cancel will disable you logging i.e. might be you might have to reconnect yourself. Whereas the Close button will leave you unchanged with what you are doing at the moment.


But in many cased the functionality of both are same.
With my limited knowlege i feel, The only difference between 'Cancel' and 'Close' button is

Cancel button: Developer writes a seperate code for it to do its job
Close button (X) : Developer just needs to write the code for the dialog box.
The (x) comes with it.

Q.If "Cancel" and "Close" produce the same result to end user, can one of them be removed?

I think that depends on the user.Some users who are less familiar with computers could find a seperate 'Cancel' button more user friendly.

Q.Why would a user prefer to have both?
User may prefer not to have a (X) button , but is it possible to remove it from the dialog box ? i dont think so..