"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Wednesday, July 30, 2008

Most test cases fail

  • Most testers who write test cases carry a hope that most test cases they write would help in finding most bugs.
  • Most testers who execute them carry a hope that most test cases they execute would help in finding most bugs.
  • Most testers who add more test cases carry a hope that they will help in finding most bugs.
  • Most customers insist on most of the testing to be done from test cases.
  • Most managers who ask testers to write test cases carry a hope that their testers will catch most bugs with the help of test cases.
  • Most managers who asked them to write test cases ask a question: How many test cases passed? when considering to ship.
  • Most testers in those contexts reply, "Most".

"If a test case didn't help in finding a bug then the purpose of that test case failed. Products are shipped when most test cases fail their purpose of existence."

------------------------------------------------------------------------------------------------------------------------------------------

  • Test cases fail to help most testers who write and execute them realize that testing is not a monotonous job as they are doing.
  • Test cases fail to help most testers realize that there are other ways to achieve what they are trying to achieve or there are other ways to do much better testing.
  • Test cases fail to help most managers understand that the number of test cases passed is a misleading metric because most test cases means few documented tests from a possible hundred billion tests.
  • Test cases fail to help most customers understand that there are ways to gain more value for the cost they are paying.
  • Test cases fail to help testers earn more money because in scripted testers view - an expert tester is one who writes a lot of test cases and executes a lot of test cases.
Test cases are human ideas and all human ideas can fail. Most testers who write and execute it fail to realize it.

If human ideas to test can't fail so will the software, not fail. Don't test.

------------------------------------------------------------------------------------------------------------------------------------------

To participate ( or just enjoy ) in a discussion involving exploratory and scripted testers on test cases , click here .

-- Pradeep Soundararajan - http://testertested.blogspot.com - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --


Update: The comments posted as Angel and Devil are by me ( Pradeep Soundararajan ). Wondering why I posted comments on the post I wrote - you would know the answer if you go through those comments.

The Angels and Devils concept in Testing is inspired by a presentation from Michael Bolton and Jonathan Kohl . Here is an excerpt of the presentation and also illustrates Elizabeth Kohl's design skills.

44 comments:

Devil said...

Most testers who write test cases carry a hope that most test cases they write would help in finding most bugs.

Well, but there are a lot of scripted testers who write test cases to prove to the customers that systems works in a correct way, so you are completely wrong.

Angel said...

Devil said Well, but there are a lot of testers who write test cases to prove if the systems works in a correct way, so you are completely wrong.

So those testers have a definition of testing as “Proving the system correct”. That’s interesting, I think there are thousands of tests that I can write to prove a system works.

I think some developers are able to prove their code works and a tester also saying “it works” is a little funny.

Devil said...

Angel said So those testers have a definition of testing as “Proving the system correct”. That’s interesting, I think there are thousands of tests that I can write and perform to prove a system works.

A part of testers job is to prove the system works as per the end user needs.

Angel said...

Devil said A part of testers job is to prove the system works as per the end user needs.

Which end user are you talking about?

If I am testing a mobile phone that is supposed to be sold in Indian subcontinent. Who are my end users and how can I think from their perspective?

I am aware that prostitutes in India also use mobile phone. Can I as a tester think from a prostitute’s view of using mobile phone?

Should I do my tests to please them?

devil said...

Angel said, Which end user are you talking about? If I am testing a mobile phone that is supposed to be sold in Indian subcontinent. Who are my end users and how can I think from their perspective?

I am aware that prostitutes in India also use mobile phone. Can I as a tester think from a prostitute’s view of using mobile phone?
Should I do my tests to please them?



Did you know? Testing is not about pleasing people.

angel said...

Devil said, Did you know? Testing is not about pleasing people.

Oh yes, it might not be about pleasing people but if you don’t find security problems, hackers get pleased of your work ( and the developer who did not fix it if you reported it and he chose not to fix )

devil said...

Angel said, Oh yes, its not about pleasing people but if you don’t find security problems, hackers get pleased of your work ( and the developer who did not fix it if you reported it and he chose not to fix )

Who said scripted testers don’t find security problems?

angel said...

Devil said, Who said scripted testers don’t find security problems?

Do you know what security problems exist and tests to find them when you are writing test cases even before you start testing the product?

devil said...

Angel said, Do you know what security problems exist and tests to find them when you are writing test cases even before you start testing the product?

So, how do you exploratory testers know that?

angel said...

Devil said, So, how do you exploratory testers know that?

We are more likely to find it because we are cognitively engaged with the product and learn it much faster than scripted testers.

As we do more tests, we know more about the product.

Limiting the tests a tester runs also limits his learning about the product.

devil said...

Angel said,Limiting the tests a tester runs also limits his learning about the product.

Do you mean to say you run all possible tests?

angel said...

devil said
Do you mean to say you run all possible tests?


I mean as we don't spend time documenting tests we have that time available to run tests.

In scripted approaches to test that I have witnessed, been a part of, and hear about a couple of months are spent on writing test cases and getting it reviewed.

If in those couple of months were more tests run, how much of valuable information and learning of the product could be gained.

devil said...

Angel said,
In scripted approaches to test that I have witnessed, been a part of, and hear about a couple of months are spent on writing test cases and getting it reviewed.

If in those couple of months were more tests run, how much of valuable information and learning of the product could be gained.


This is funny. Writing test cases and getting it reviewed and approved are a part of process we follow to ensure quality.

angel said...

devil wrote This is funny. Writing test cases and getting it reviewed and approved are a part of process we follow to ensure quality.

Oh, you ensure quality. So is quality about those five thousand test cases passing?

If I ask the customers you serve on what quality means to them - would they say - "If 5000 test cases pass then I would call it quality" or would they look at the value the product adds to them for the price they pay?

devil said...

angel wrote,
If I ask the customers you serve on what quality means to them - would they say - "If 5000 test cases pass then I would call it quality" or would they look at the value the product adds to them for the price they pay?


We cover most of things that we think what our customers would call as quality.

angel said...

devil said,
We cover most of things that we think what our customers would call as quality.


I have two questions:

1. If you have the capability to cover "most", why not cover the rest that can be termed "complete".

2. Have you ever asked your customer what he means by quality?

devil said...

angel wrote
1. If you have the capability to cover "most", why not cover the rest that can be termed "complete".


For which I would answer: No one can do a complete testing and I am surprised you don't know that.

2. Have you ever asked your customer what he means by quality?

Do you think customers would give us project and expect to ask such a silly thing?

angel said...

devil asked:


For which I would answer: No one can do a complete testing and I am surprised you don't know that.


Hmm! My question was when you guys know "most" you must know what "complete" is to achieve "most".

For instance if I am racing in a Formula 1 car and I have lapped the circuit 49 out of 60 laps I am supposed to complete, I might be in a position to say, "most of my laps are done". If I didn't know what completing the laps means, would I ever be in a position to say "most"?

Do you think customers would give us project and expect to ask such a silly thing?

How silly might the customer think of you for not asking this question and saying, "as 5000 test cases we wrote passed during execution we say it's good quality"?

devil said...

Angel,

You must understand that is a team of 5 people are asked to explore then the coverage they achieve would not be known.

angel said...

devil wroteYou must understand that is a team of 5 people are asked to explore then the coverage they achieve would not be known.

I observe a common pattern: All those who talk against scripted testing are the ones who have burned their hands and have witnessed projects getting burned by scripted approach and those who talk against exploratory testing aren't someone who practiced exploratory testing as seriously as it should be.

Moreover I might suggest you to look into things like Session Based Test Management at www.satisfice.com to know how to achieve coverage through exploratory testing.

What do you think exploratory testing is?

devil said...

Exploratory testing is about finding bugs by playing with the product or its ad hoc about going here and there based on the experience and finding bugs.

angel said...

devil wrote Exploratory testing is about finding bugs by playing with the product or its ad hoc about going here and there based on the experience and finding bugs.

Ah! So its this idea that makes you say whatever you are saying.

How about looking at the idea of exploratory testing from the man who coined it ( Cem Kaner ):

“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

devil said...

Ooooh! you guys come out with some idea and want me to follow it.

angel said...

devil wrote Ooooh! you guys come out with some idea and want me to follow it.

So, is scripted testing or test case based approach to testing ideas that you came out with?

devil said...

It might not be my idea but it is giving success to many people.

angel said...

devil said, It might not be my idea but it is giving success to many people

How do you define success in testing?

devil said...

Success of testing is when product meets the specification.

angel said...

Success of testing is when product meets the specification.

The "incomplete" specification you mean?

devil said...

If a specification isn't complete then its not our head ache.

angel said...

If a specification isn't complete then its not our head ache.

Escapism is true identity of a scripted tester which you demonstrated by saying that.

By being an exploratory tester I am more responsible for the questions I ask, the tests I do and the report of my findings.

By being a scripted tester you always tend to blame on the script for not finding a bug. Managers also say, "Why did we miss that test case?" whenever a bug slips out.

It might be a good idea to ask: What skills had we had might have helped us find this bug within our facility?

devil said...

Pradeep wrote "If a test case didn't help in finding a bug then the purpose of that test case failed. Products are shipped when most test cases fail their purpose of existence."

He is an idiot. Test cases help in educating new testers who come on board to the project.

angel said...

Test cases help in educating new testers who come on board to the project.

I have three questions:

1. Is the mission of testing: To educate testers who join in between the product development OR To find information about the quality of the product?

2. I agree that the education part is important but how about recording testing activity and presenting a video. For instance Test Explorer ( a tool developed by David Gilbert ) records testing activity done on windows based products and throws out a video. How about that?

3. If I were to educate you on how to fly an aeroplane, would you prefer to learn by flying the aeroplane with someone experienced sitting as a co-pilot or by flying with scripts which say,

Step One: Turn on the engine
Step Two: Increase the throttle
Step Three: Run in the runway
Step Four: Take Off

devil said...

Do you think people on project have time to sit as co pilot and teach the ones who joined new how to test?

angel said...

Do you think people on project have time to sit as co pilot and teach the ones who joined new how to test?

If they don't have time I am surprised how they wasted time writing thousands of test cases thinking that someone would join in between the project and it would benefit them.

devil said...

I dont think its any use arguing with you and I think I wasted time.

angel said...

I dont think its any use arguing with you and I think I wasted time.

That's precisely the point I am making. Scripted testers spend too much time on what is not worth - writing test cases and executing them

devil said...

I think you are a devil to testing because you are bringing such impractical ideas to testing.

angel said...

You are right. Angel and Devils are perspective. You appear as a devil to me and I appear as a devil to you.

So are there no angels in testing?

devil said...

I don't know if there are angels in testing but all I know is you are a devil.

angel said...

If you don't know about Angels in testing nor their work nor the way they test, you would never become an angel or you would never know you became an angel.

I can show of an Angel I know: James Bach

I can challenge you that he will be able to challenge, test and demonstrate the value that he can add as a tester.

Can you from your scripted approach show one Angel who can challenge James Bach in testing?

devil said...

Why should I show you an Angel from Scripted approach?

angel said...

To let me know if they exist.

Pradeep Soundararajan said...

@Angel and Devil,

Thanks for this on going discussion on test cases, philosophy of scripted testing, exploratory testing.

I would suggest that we hear from other people as well and you guys stop posting comments for unless you see a need for it.

Manoj said...

Thanks Mr. Angel and Mr.Devil, your conversation/debate was very informative and you have talked on lots of topics upon which a "scripted tester" and an "exploratory tester" debate on all the time :-)

This article is a class apart from all other articles which I have read on this site or on other sites. Pradeep you have put in an excellent effort to give direction and explanations to this debate. Your Devil and Angel characters were great. It seemed that you can go on for ever with your advocacy on exploratory testing.

And obviously I got answers to many questions which I had in mind and now I can visualize an Exploratory Teseter in a much clearer way.

Pradeep, looking forward to more articles involving Mr. Angel and Mr.Devil.