- Most testers who write test cases carry a hope that most test cases they write would help in finding most bugs.
- Most testers who execute them carry a hope that most test cases they execute would help in finding most bugs.
- Most testers who add more test cases carry a hope that they will help in finding most bugs.
- Most managers who ask testers to write test cases carry a hope that their testers will catch most bugs with the help of test cases.
- Most managers who asked them to write test cases ask a question: How many test cases passed? when considering to ship.
- Most testers in those contexts reply, "Most".
"If a test case didn't help in finding a bug then the purpose of that test case failed. Products are shipped when most test cases fail their purpose of existence."
------------------------------------------------------------------------------------------------------------------------------------------
- Test cases fail to help most testers who write and execute them realize that testing is not a monotonous job as they are doing.
- Test cases fail to help most testers realize that there are other ways to achieve what they are trying to achieve or there are other ways to do much better testing.
- Test cases fail to help most managers understand that the number of test cases passed is a misleading metric because most test cases means few documented tests from a possible hundred billion tests.
- Test cases fail to help most customers understand that there are ways to gain more value for the cost they are paying.
- Test cases fail to help testers earn more money because in scripted testers view - an expert tester is one who writes a lot of test cases and executes a lot of test cases.
Test cases are human ideas and all human ideas can fail. Most testers who write and execute it fail to realize it.
If human ideas to test can't fail so will the software, not fail. Don't test.
If human ideas to test can't fail so will the software, not fail. Don't test.
------------------------------------------------------------------------------------------------------------------------------------------
To participate ( or just enjoy ) in a discussion involving exploratory and scripted testers on test cases , click here .
-- Pradeep Soundararajan - http://testertested.blogspot.com - pradeep.srajan@gmail.com
"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --
Update: The comments posted as Angel and Devil are by me ( Pradeep Soundararajan ). Wondering why I posted comments on the post I wrote - you would know the answer if you go through those comments.
The Angels and Devils concept in Testing is inspired by a presentation from Michael Bolton and Jonathan Kohl . Here is an excerpt of the presentation and also illustrates Elizabeth Kohl's design skills.
44 comments:
Most testers who write test cases carry a hope that most test cases they write would help in finding most bugs.
Well, but there are a lot of scripted testers who write test cases to prove to the customers that systems works in a correct way, so you are completely wrong.
Devil said Well, but there are a lot of testers who write test cases to prove if the systems works in a correct way, so you are completely wrong.
So those testers have a definition of testing as “Proving the system correct”. That’s interesting, I think there are thousands of tests that I can write to prove a system works.
I think some developers are able to prove their code works and a tester also saying “it works” is a little funny.
Angel said So those testers have a definition of testing as “Proving the system correct”. That’s interesting, I think there are thousands of tests that I can write and perform to prove a system works.
A part of testers job is to prove the system works as per the end user needs.
Devil said A part of testers job is to prove the system works as per the end user needs.
Which end user are you talking about?
If I am testing a mobile phone that is supposed to be sold in Indian subcontinent. Who are my end users and how can I think from their perspective?
I am aware that prostitutes in India also use mobile phone. Can I as a tester think from a prostitute’s view of using mobile phone?
Should I do my tests to please them?
Angel said, Which end user are you talking about? If I am testing a mobile phone that is supposed to be sold in Indian subcontinent. Who are my end users and how can I think from their perspective?
I am aware that prostitutes in India also use mobile phone. Can I as a tester think from a prostitute’s view of using mobile phone?
Should I do my tests to please them?
Did you know? Testing is not about pleasing people.
Devil said, Did you know? Testing is not about pleasing people.
Oh yes, it might not be about pleasing people but if you don’t find security problems, hackers get pleased of your work ( and the developer who did not fix it if you reported it and he chose not to fix )
Angel said, Oh yes, its not about pleasing people but if you don’t find security problems, hackers get pleased of your work ( and the developer who did not fix it if you reported it and he chose not to fix )
Who said scripted testers don’t find security problems?
Devil said, Who said scripted testers don’t find security problems?
Do you know what security problems exist and tests to find them when you are writing test cases even before you start testing the product?
Angel said, Do you know what security problems exist and tests to find them when you are writing test cases even before you start testing the product?
So, how do you exploratory testers know that?
Devil said, So, how do you exploratory testers know that?
We are more likely to find it because we are cognitively engaged with the product and learn it much faster than scripted testers.
As we do more tests, we know more about the product.
Limiting the tests a tester runs also limits his learning about the product.
Angel said,Limiting the tests a tester runs also limits his learning about the product.
Do you mean to say you run all possible tests?
devil said
Do you mean to say you run all possible tests?
I mean as we don't spend time documenting tests we have that time available to run tests.
In scripted approaches to test that I have witnessed, been a part of, and hear about a couple of months are spent on writing test cases and getting it reviewed.
If in those couple of months were more tests run, how much of valuable information and learning of the product could be gained.
Angel said,
In scripted approaches to test that I have witnessed, been a part of, and hear about a couple of months are spent on writing test cases and getting it reviewed.
If in those couple of months were more tests run, how much of valuable information and learning of the product could be gained.
This is funny. Writing test cases and getting it reviewed and approved are a part of process we follow to ensure quality.
devil wrote This is funny. Writing test cases and getting it reviewed and approved are a part of process we follow to ensure quality.
Oh, you ensure quality. So is quality about those five thousand test cases passing?
If I ask the customers you serve on what quality means to them - would they say - "If 5000 test cases pass then I would call it quality" or would they look at the value the product adds to them for the price they pay?
angel wrote,
If I ask the customers you serve on what quality means to them - would they say - "If 5000 test cases pass then I would call it quality" or would they look at the value the product adds to them for the price they pay?
We cover most of things that we think what our customers would call as quality.
devil said,
We cover most of things that we think what our customers would call as quality.
I have two questions:
1. If you have the capability to cover "most", why not cover the rest that can be termed "complete".
2. Have you ever asked your customer what he means by quality?
angel wrote
1. If you have the capability to cover "most", why not cover the rest that can be termed "complete".
For which I would answer: No one can do a complete testing and I am surprised you don't know that.
2. Have you ever asked your customer what he means by quality?
Do you think customers would give us project and expect to ask such a silly thing?
devil asked:
For which I would answer: No one can do a complete testing and I am surprised you don't know that.
Hmm! My question was when you guys know "most" you must know what "complete" is to achieve "most".
For instance if I am racing in a Formula 1 car and I have lapped the circuit 49 out of 60 laps I am supposed to complete, I might be in a position to say, "most of my laps are done". If I didn't know what completing the laps means, would I ever be in a position to say "most"?
Do you think customers would give us project and expect to ask such a silly thing?
How silly might the customer think of you for not asking this question and saying, "as 5000 test cases we wrote passed during execution we say it's good quality"?
Angel,
You must understand that is a team of 5 people are asked to explore then the coverage they achieve would not be known.
devil wroteYou must understand that is a team of 5 people are asked to explore then the coverage they achieve would not be known.
I observe a common pattern: All those who talk against scripted testing are the ones who have burned their hands and have witnessed projects getting burned by scripted approach and those who talk against exploratory testing aren't someone who practiced exploratory testing as seriously as it should be.
Moreover I might suggest you to look into things like Session Based Test Management at www.satisfice.com to know how to achieve coverage through exploratory testing.
What do you think exploratory testing is?
Exploratory testing is about finding bugs by playing with the product or its ad hoc about going here and there based on the experience and finding bugs.
devil wrote Exploratory testing is about finding bugs by playing with the product or its ad hoc about going here and there based on the experience and finding bugs.
Ah! So its this idea that makes you say whatever you are saying.
How about looking at the idea of exploratory testing from the man who coined it ( Cem Kaner ):
“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”
Ooooh! you guys come out with some idea and want me to follow it.
devil wrote Ooooh! you guys come out with some idea and want me to follow it.
So, is scripted testing or test case based approach to testing ideas that you came out with?
It might not be my idea but it is giving success to many people.
devil said, It might not be my idea but it is giving success to many people
How do you define success in testing?
Success of testing is when product meets the specification.
Success of testing is when product meets the specification.
The "incomplete" specification you mean?
If a specification isn't complete then its not our head ache.
If a specification isn't complete then its not our head ache.
Escapism is true identity of a scripted tester which you demonstrated by saying that.
By being an exploratory tester I am more responsible for the questions I ask, the tests I do and the report of my findings.
By being a scripted tester you always tend to blame on the script for not finding a bug. Managers also say, "Why did we miss that test case?" whenever a bug slips out.
It might be a good idea to ask: What skills had we had might have helped us find this bug within our facility?
Pradeep wrote "If a test case didn't help in finding a bug then the purpose of that test case failed. Products are shipped when most test cases fail their purpose of existence."
He is an idiot. Test cases help in educating new testers who come on board to the project.
Test cases help in educating new testers who come on board to the project.
I have three questions:
1. Is the mission of testing: To educate testers who join in between the product development OR To find information about the quality of the product?
2. I agree that the education part is important but how about recording testing activity and presenting a video. For instance Test Explorer ( a tool developed by David Gilbert ) records testing activity done on windows based products and throws out a video. How about that?
3. If I were to educate you on how to fly an aeroplane, would you prefer to learn by flying the aeroplane with someone experienced sitting as a co-pilot or by flying with scripts which say,
Step One: Turn on the engine
Step Two: Increase the throttle
Step Three: Run in the runway
Step Four: Take Off
Do you think people on project have time to sit as co pilot and teach the ones who joined new how to test?
Do you think people on project have time to sit as co pilot and teach the ones who joined new how to test?
If they don't have time I am surprised how they wasted time writing thousands of test cases thinking that someone would join in between the project and it would benefit them.
I dont think its any use arguing with you and I think I wasted time.
I dont think its any use arguing with you and I think I wasted time.
That's precisely the point I am making. Scripted testers spend too much time on what is not worth - writing test cases and executing them
I think you are a devil to testing because you are bringing such impractical ideas to testing.
You are right. Angel and Devils are perspective. You appear as a devil to me and I appear as a devil to you.
So are there no angels in testing?
I don't know if there are angels in testing but all I know is you are a devil.
If you don't know about Angels in testing nor their work nor the way they test, you would never become an angel or you would never know you became an angel.
I can show of an Angel I know: James Bach
I can challenge you that he will be able to challenge, test and demonstrate the value that he can add as a tester.
Can you from your scripted approach show one Angel who can challenge James Bach in testing?
Why should I show you an Angel from Scripted approach?
To let me know if they exist.
@Angel and Devil,
Thanks for this on going discussion on test cases, philosophy of scripted testing, exploratory testing.
I would suggest that we hear from other people as well and you guys stop posting comments for unless you see a need for it.
Thanks Mr. Angel and Mr.Devil, your conversation/debate was very informative and you have talked on lots of topics upon which a "scripted tester" and an "exploratory tester" debate on all the time :-)
This article is a class apart from all other articles which I have read on this site or on other sites. Pradeep you have put in an excellent effort to give direction and explanations to this debate. Your Devil and Angel characters were great. It seemed that you can go on for ever with your advocacy on exploratory testing.
And obviously I got answers to many questions which I had in mind and now I can visualize an Exploratory Teseter in a much clearer way.
Pradeep, looking forward to more articles involving Mr. Angel and Mr.Devil.
Post a Comment