Most test cases fail

• Most testers who write test cases carry a hope that most test cases they write would help in finding most bugs.

• Most testers who execute them carry a hope that most test cases they execute would help in finding most bugs.

• Most testers who add more test cases carry a hope that they will help in finding most bugs.

• Most customers insist on most of the testing to be done from test cases.

• Most managers who ask testers to write test cases carry a hope that their testers will catch most bugs with the help of test cases.

• Most managers who asked them to write test cases ask a question: How many test cases passed? when considering to ship.

• Most testers in those contexts reply, "Most".

"If a test case didn't help in finding a bug then the purpose of that test case failed. Products are shipped when most test cases fail their purpose of existence."

------------------------------------------------------------------------------------------------------------------------------------------

• Test cases fail to help most testers who write and execute them realize that testing is not a monotonous job as they are doing.
  

• Test cases fail to help most testers realize that there are other ways to achieve what they are trying to achieve or there are other ways to do much better testing.

• Test cases fail to help most managers understand that the number of test cases passed is a misleading metric because most test cases means few documented tests from a possible hundred billion tests.
  

• Test cases fail to help most customers understand that there are ways to gain more value for the cost they are paying.

• Test cases fail to help testers earn more money because in scripted testers view - an expert tester is one who writes a lot of test cases and executes a lot of test cases.

Test cases are human ideas and all human ideas can fail. Most testers who write and execute it fail to realize it.

If human ideas to test can't fail so will the software, not fail. Don't test.

------------------------------------------------------------------------------------------------------------------------------------------

To participate ( or just enjoy ) in a discussion involving exploratory and scripted testers on test cases , click here .

-- Pradeep Soundararajan - http://testertested.blogspot.com - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --


Update: The comments posted as Angel and Devil are by me ( Pradeep Soundararajan ). Wondering why I posted comments on the post I wrote - you would know the answer if you go through those comments.

The Angels and Devils concept in Testing is inspired by a presentation from Michael Bolton and Jonathan Kohl . Here is an excerpt of the presentation and also illustrates Elizabeth Kohl's design skills.

Heroes of software testing - Do you know about them and their work?

My initial plan was to board the flight to Toronto on July 11th from Mumbai to attend CAST 08 software testing conference. Sangeeta, a tester from Mumbai influenced me to change my plan and organized a lecture in Ness Technologies in Mumbai.

I liked the offer and accepted it, as I knew I would enjoy the experience. It's fun to meet new testers, talk to them and learn new things. I am in constant search for budding testers like Sharath Byregowda , Ajay Balamurugadas , Sathish Kumar Chinappa , Sangeeta, Sandesh , Girija, Bhargavi and Nattu ( who accompanied me to CAST 08 ) and those who are passionate and do things about seeing a better tester community like Mohan Panguluri, COO of Edista Testing ... I never know where I can find similar ones. One of the ways that has been of great help to me, to find such testers and thinkers is, while at lectures and my workshops. I offer whatever help I am capable of doing to expose them to good stuff and I do it when they ask me for it.

Here is an excerpt from the talk:

"We are in India, the land where cricket is considered close to being a religion. It has super heroes like Sachin Tendulkar and Dhoni who have inspired a lot of people in India to play better cricket. They could inspire because a lot of people spent their time watching them play great cricket at tough situations. As Indians, you might have seen a lot of people who enjoy playing cricket the way Sachin and Dhoni play. As testers, have you seen anyone doing testing that has inspired you to test better?"

Response: Silence

and then a question comes up from a corner, "Who did Sir Don Bradman see as an inspiration for he is the best known cricketer of centuries?" and all other testers burst to laughter thinking that I was cornered with that question.

I took a while for the audience to settle down and replied,
"I know who inspired Sir Don Bradman" and then a pause to add, "It was himself."
"How many of you are inspired by your own testing?"

Response: Silence

End of excerpt

Reading good stuff as a bad practice and reading bad stuff as the best practice

The above state is what most Indian testers to my knowledge are stuck in. This state resulted because most of testers don't have the habit of reading as though its a bad practice. Most of those who read, are the ones who search in Google and land up at bad sites ( that includes my blog at times ). Most of those who Google search are the ones who are not interested at better thought process but at ready made best practice answers.

There are good and great stuff from internet. I once searched for expert testers and found James Bach. That google search changed my life and the way I test. There were a lot of junk stuff written to attract such search results that I had to pass through before finding James Bach's website. I was sure that I needed help about thought process from an expert tester and not ready made solutions from time to time.

The road to Toronto

I had to face a lot of trouble to get to Toronto. My Visa was rejected ( at the first attempt ), I didn't have sufficient funds ( as per the VISA officials and my bank statement ) , I was denied permission to board the plane as my ticket was booked via United States and I didn't have a US Visa. The Canadian customs had a discussion that bothered me if they would let me in after I landed at Toronto and then my baggage went missing and finally landed as the last one on the belt. Amidst all this was my hope to meet the heroes in real space who have inspired me and to meet those whom I can draw more inspiration from. I believe in God and also believe that God takes a human form to help humans. A great human who lives in 61, Ashburnham in Toronto and the one who lived in my house ( my dad ) helped me and cleared traps so that I reached Toronto and was able to attend CAST 08.


Chasing dreams

Finally I got a chance to meet the heroes who have been inspiring me to test and think better. My super hero James Bach wasn't present at the conference and hence I couldn't meet him. I would be meeting him in November at a conference where we both are invited to speak on testing at a developers conference in Malmo Sweden.

By meeting my heroes, I learned some important things that I could not have learned, had I not met them. I got an insight into the way they live, some ways they think and they communicate. It was several dreams of several years that came true, all in one place - CAST 08.

Not all learning is fun.

If you have understood what I have written in this post so far, then my English writing skill has improved a lot since I wrote my first blog post. It pained a lot to know that I was writing terribly bad English but had I not learned it, I wouldn't have been able to make sense to you in this post.

One of the important things I learned in CAST is the cost of making a few mistakes and the cost of wasting someone's time. I did some mistakes a few months ago but experienced the cost of making those mistakes when I could not stand in front of those people whose time I had wasted. I felt too ashamed of myself but I know I would not feel ashamed for a life time as I am recovering from those mistakes.

Learning is fun as long as you don't learn about yourself and how bad you are in somethings that you want to be really good at. The more I learn that I am bad, I am inspired to work hard enough to get over it. Being a student of James Bach means I keep searching for the answer, "How do I know what I know?" and often I discover what I don't know when I meet and discuss with great minds like the ones I met in CAST 08. I make it a practice to learn what I don't know or what more I'd want to learn on what I want to be good at.


Not all pain is bad

I went all way from Bengaluru ( Bangalore ) to Toronto to know more about myself and how bad I am in things I want to be good at. It hurts but there is no short cut to learning. For those who might disagree that learning could hurt - as testers we provide information about the quality of the product to the management. If we find a lot of bugs, it might hurt the management when they know it especially when it upsets their shipping schedules. It hurts them because they have learned something about their own product. This pain is then converted to measures to not get into a similar situation again. That is what I think some great man said, "No pain, No gain". The things that I learned from my heroes at the conference and other discussions would be reflected in my blog for years to come, so unfortunately keep reading them.


Person dependent software projects and organization

India depended on Sachin Tendulkar for winning a lot of matches. Cricket ( a team sport ) became a person dependent one. Is it a problem for Sachin?

Not really. Actually it demanded him to be at a very good knock and he loved that challenge as it challenged its consistency. It could also be stated that - he faced tough situations and cleared balls beyond boundaries which is why he is admired by millions worldwide.

Similarly, if your organization doesn't want to make their testing a person dependent thing it is because they don't like you to be Sachin Tendulkar . They can't stop from you becoming a Sachin of software testing because it depends on your learning as a tester and not they giving a learning opportunity to you.

You will be interested to read what James Bach wrote about the need for super heroes for software projects long back.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

It's a "tester" who finds a bug, even with the robust Google Search

The image you might see above is a screen shot of what I saw after hitting the 12th page for the search results of "tester". I am not sure if you can reproduce that because I haven't investigated on it but I did plan to capture it to demonstrate:

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it."

Be it with the robust systems like Google Search Engine or with weak systems that we might be using, it is always a HUMAN who finds a bug. A lot of testers I know think of "test case" finding a bug.

There is a test case document that consists of 9856985956895869698569956985698459698 test cases and no tester executing it wouldn't find bugs by itself. There is a test case document with 3 documented tests and a tester takes the help of that to find bugs when he executes, observes the result and recognizes a bug.

A test case is an extension of a test idea. What matters to a tester is a test idea and not the test case. Skilled (exploratory testers ( humans ) use tons of ideas ( heuristics and oracles ) to find and recognize bugs. That's why they can find more bugs that matter than those running thousands of test cases over and over again.

Honestly, 99% of testers I have come across didn't say - "I read each test case each time I have to execute it after I have done it once. Also, I religiously follow what is written in the test case".

What happens when they deviate from the documented test case is, they are exploring and running different test. Maybe they don't like to call it that way because their management who pays wouldn't like to know that they are not executing the "test cases".

That's how customers are fooled by management saying "yes, we are running test cases" and yet benefited by the testing community by running more tests.

A test idea can be executed in hundreds of different ways. Check out the deep analysis made by James Bach and Michael Bolton on What Do Scripts Tell Us?


--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

The (bad) state of software testing interviews in India

Alternatively, you can click on this link

Letter to myself

Dear Pradeep,

Greetings!

I have been with you all this while and shall continue to be with you. I am enjoying each moment you enjoy as a tester and also each moment that you don't enjoy as a tester. I am sorry for enjoying those moments which you think you haven't been enjoying. I am sure you'd be interested to know why and how I have been enjoying those moments that you haven't been.

First, let me list the thing that you haven't been enjoying:

• Whenever and wherever you write about test automation, a handful of readers tend to think that you reject the idea of test automation and they write harsh e-mails to you.
• Whenever and wherever you write about certification, the certified tester community attacks you over e-mails/phone calls that you and I are spoiling the craft.
• Whenever and wherever you write about programming skill for testers, another handful readers think that you are suggesting testers not to learn programming.
• Whenever and wherever you write about tools, most testers think you are referring to test automation tool.
• Whenever and wherever you write about yourself or your experience, a set of people think you lack humility.
• Whenever and wherever you write about exploratory testing as a skilled activity, a set of people think "no tester would do be able to do that".
• Whenever and wherever you write about ideas to solve a problem than giving a one line answer, a set of people think you don't know how to solve it and is faking what you know.
• Whenever and wherever you are writing about testing being an excellent thinking job, a few people think you are trying to paint a picture that does not exist.
• Whenever and wherever you - do bad testing, fail in testing course like BBST, you feel intimidated by more skilled people than you, you feel bad about not having learned or practiced those things that helps you become a better tester, you fail to give enough respect to expert testers time, etc...
  

In this context, I'd like to remind you of a learning you had from Michael Bolton: There are some things under your control and there are other things that are not under your control. Taking advantage of things under your control, as a tester, is essential to clear traps and it might also lead to gaining more control. To take advantage of things under your control, you first should realize what are the things you control.

I also remember that you had made a note in your Moleskine of Saurav Ganguly's television interview where he was asked: How were you able to make a great comeback to the world cup cricket squad after being axed for poor performance? His answer: I didn't worry about things that are not under my control ( media critique, jokes on bad batting performance, e-mail forwards about my performance, people gossiping about it ) but focussed on things that are under my control ( Practice, skill enhancement, consistent batting record in Ranji trophy)

Similarly, you don't have a control over the thoughts of people thinking whatever they think after reading whatever you have written. You have a control over what you write and you have a control over the way you write it.

Your testing has been influenced by a lot of experts but not all have similar influence. They haven't seen great testing to appreciate the things you are sharing and I doubt if all those who witness it would be influenced by it because it's hard work and high skill demanding.

• Not all testers want to do great testing
  
• Not all testers know they are doing bad testing.
• Not all testers want to know they are doing bad testing.
• Not all testers want to know more about testing.
• Not all testers know what skills to gain and practice.
• Not all testers agree to be context driven.

Here are three questions ( like the Monty Python and the Holy grail bridge of death piece you enjoy )

1.Whom are you serving through your writing?

I am sure your ongoing struggle is in understanding that. Let me help you with what I think about whom you are serving - You are serving those testers who look for better thought process and those who enjoy the better thought process and those who think you have a better thought process.

2. Who asked you to serve them?

I asked you to do that!

3. Why haven't you been enjoying some moments that I have been?

You want all testers to do great testing although you know its not possible. Some people question your idea of "great testing" because they already have an idea of "great testing" and it conflicts with the idea you have. You are able to demonstrate that their idea of "great testing" lacks critical thought as your idea of "great testing".

By the way, your idea of "great testing", is not yours but of those people who have influenced you. You have just subscribe to those ideas and are contributing to it in different forms. I have occasionally witnessed you doing bad testing and I am sure I would see that in future, too. Do not forget that you are a human and your ideas are fallible. I know bad testing and bad thought process irritates you, even if you are the one who is doing it.

I would love to see you doing things that are under your control - learning, reading, writing, bettering your skills, helping those testers who enjoy the thought process that you enjoy, speaking, coaching and mentoring.

Your power to influence testers is limited. Limited to the ones who don't want to limit themselves. So do unlimited things under limited time that you and I will be here in this world.

I will be with you forever, enjoying everything you do from great things to not so great things. Anything you do is great to me.

I will write to you whenever I feel a need for it. This letter is personal, just between you and me.

"Here is a way to test if your mission on earth is complete - if you are alive, it isn't" -- Richard Bach

Yours truly,

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Reduce Reuse Recycle

I toured Singapore for about 10 days this month. I don't know why my eyes kept catching signboards that suggested people to Reduce, Reuse and Recycle plastics. I think it was because I am from India.

As a part of my tour, I had been to a Zoo and there, I heard the animal show host explaining the need to reduce, reuse and recycle as it impacts the environment and animals.

I felt great about Singaporeans, for stressing the point in many places. No wonder their country ( most parts ) is so clean and tidy that I felt I am in a foreign land. I thought for a while that people in India don't talk about Reduce, Reuse and Recycle as much as Singaporeans do.

16 days after the tour ended, I realized I was wrong. People in India talk as much as Singaporeans do about reducing, reusing and recycling. Here is how they ask:

How can I reduce testing and thereby decrease the cost of the project and increase my profitability?

If testing is a job of providing quality related information to the stake holders to help them take better informed what could reducing testing mean?

I think the people who talk about reducing testing to decrease costs want to know if there is

• a way (or more than one) to make testers more efficient, skilled and competent.
• a way (or more than one) to get information more faster.
• a way (or more than one) to know if they can save costs by not purchasing unnecessary tools.
• a way (or more than one) to know if they can avoid hiring bad candidates.
• a way (or more than one) to know if they could spend less on managing attrition.
• a way (or more than one) to know if they are not doing things that would not add value.
• a way (or more than one) to know if they are meeting the mission.
• a way (or more than one) to know if they can fall into fewer traps.
• a way (or more than one) to know if they can recover much faster from the traps.
• a way (or more than one) to know if they can be more successful in getting more projects.

and : How can I reuse and recycle my test cases, test scripts and testers for future projects thereby reducing the cost of my future projects to increase my profitability?

I think people who talk about reusing / recycling test cases, scripts and testers for future projects want to know:

• many ways (or at least one) to know if they can cut costs by not needing to train testers on a similar domain, technology or product.
  
• many ways (or at least one) to know if they can cut costs by not having to spend time on writing test cases and test scripts for a new project by modifying existing test cases and test scripts.
• many ways (or at least one) to know if they can cut costs by not having to write code that tests code by modifying existing code.
• many ways (or at least one) to know if they can cut costs by not needing to spend much of a time on learning the new product and finding bugs faster.
• many ways (or at least one) to know if by thinking of reusing and recycling, they are definitely saving costs without sacrificing the value they want to add.

Well, if all those who ask the questions knew how to ask it more elaborate or deeper questions, we'd be living in a different world. Thankfully, we still are in the same world.

How did I reuse, recycle and reduce?

• I don't recommend writing test cases and executing tests with the help of that. Although I am not someone who'd like looking at test cases, there was a context in which I looked in to test case document that someone else had written, to gather ideas for my exploratory testing. That's how I reused a test case. It definitely reduced the cost because I took help of an already existing database of ideas. ( That doesn't mean test cases can be handy for ideas to test. A check list, cheat sheet, mnemonics, heuristics... might do it as well more cost effectively).
  

• In one of the several product development organizations I worked for, I identified that a tester was not performing fair enough. I probed for his history of performance within the organization in past projects and recommended him to be fired ( of course, I had the authority to recommend ) He was being paid a lot for he had over 7 years experience. The management feared firing him could send wrong signals to other team members but I asked, "What more right a signal can people get?". On firing him we had money to afford hiring 4 junior testers for 1/5th of what he was paid and got more than what he was delivering. That's how I reduced the cost of the product.

There might be a lot of ways to solve a problem and there might be a lot more ways to not solve them. Unfortunately the ways to not solve a problem appear like the ways to solve problem. Humans are stuck!

If working on reducing, reusing and recycling test cases aren't working, you might want to reduce your intention of reducing, reusing and recycling /and/ think of reusing those ideas in a different context /or /recycle those ideas at a later date when you think the context has changed to suit it.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Using "testing" || Abusing "testing"

As you have started to read this post, before you continue further, I'd like you to listen to a podcast - The Word Test . If you are skipping the podcast, it's OK but it might be a good idea to not skip.

Also, don't read this post while you are mid way listening to the podcast, it's more bad than not listening to it.

One of my student who works for a leading IT services provider from India, asked a question to testers in the organization he works for - "Is it good to stop testing after a couple of years of experience or after promoted to a lead or a manager?" [ My intention of this post is not to answer this question that my student asked but ... ]

There were responses like:

One need not do hands-on testing all through his/her career. When you test all by yourself, you are adding a value of say 'X' to your project. When you manage say 5 Testers, you are letting your skills, knowledge and experience on Testing percolate to 5 other members and you would be adding a value of 5 times X to your project.

and

Whatever you have said is ok for a resource working in a team which uses tools for testing. For someone who is into manual testing where is the career growth? For those wouldn't management be a blessing to be grabbed with both hands?

and

You cant be a tester for all your life. Same is the case with development. You need to manage things at one point in time. But when and where, you need to decide yourself......

and

more such.

All of these people ( including my student ) and maybe testers who are sitting nearby your cubicles while you read this mean "testing" as test execution. [ You wouldn't be surprised at this, had you listened to the podcast ]

Here is my question to those people: If testing means test execution, under what category does - test planning, test data collection, finding bugs, reporting bugs, triaging bugs, test set up, test bed creation, test documentation, thinking of test techniques, exploring, investigating bugs, reviewing test results, test reporting, modeling, diversifying test approaches, etc... fit in?

Well, when the word testing could mean so many things, why are most of us thinking only about test execution when someone uses the word "testing". This makes me question, how many people who claim to be testers really know little about testing that is enough to communicate with people without such ambiguity?

A lot of testers' only thinking is -- every thing in this field is defined pretty well and no need to think beyond it. A definition, in my opinion, should be viewed as a help for a human to think further on it and not in stopping to think beyond what it states.

In another context, if you ask them what "testing" means, they'd love to share their own impractical definitions like:

"Testing is a process of making a product bug free" OR "Testing is a systematic approach towards delivering a quality product" OR "Testing is about following quality processes to ensure bugs don't leak to customer"
and more such stupid stuff !

That's an evidence that the word "testing" itself is context sensitively used by the whole world out of which most of them might disagree with the context driven testing community about their approach. Funny world!

Ben Simo, in a recent conversation, helped me become conscious of the fact that the word "test" is both a noun and a verb; and that one feeds the other.


If one doesn't know what "testing" means, how will they ever know when they are stopping to do it?

If you think you have benefited by this post, here is a "test" you might want to take:

• What would you say, when you want to communicate that you are doing test execution?
• What would you say, when you want to communicate that you are stopping to execute tests?
• What word would you use instead of "testing" to communicate any specific activity that you do as a part of testing?
• When someone uses the word "testing", what would you want to ask them?
• When someone says "test", would you be curious to know if it is a verb or a noun?
• What would you want to know if someone said, "I want to do testing"?
  
• Would you be interested to pass this learning to someone with whom you have been communicating on "testing"?

If you think you haven't been benefited by this post, here are things you could do:

• Read it once again ;-)
• Listen to the podcast, if you have missed it ;-)
• And then exit. It's just not worth one more glance, for today.
  

As testers we use the word "testing" so many times in our life without ever (knowing) wanting to know if we abused it, too. I have done it, too. There is nothing wrong in abusing the word "testing" as long as you don't know that it means a lot by itself and in different contexts.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Graphology, Music, Chess, Cooking, Drawing and becoming a better tester

I am observing that every passionate learning tester have tried doing things that are not testing and try to bring in some learnings to testing from doing them, to test better. For instance, Michael Bolton has tried cooking and theater and has benefited by it in testing, you could read his interview in which he mentions that. Michael Hunter (a.k.a The Braidy Tester) has recently blogged about Drawing and how it has been helping him to test and think about testing. Jonathon Kohl's Exploratory Testing-Music of Investigation is no exception. Bach brothers ( James Bach and Jon Bach) are active in Chess.com and you could read how playing Chess helps to test better by reading Jon Bach's post.

CAST 08 has its theme "Beyond the Boundaries: Interdisciplinary Approaches to Software Testing". Interdisciplinary approaches drawn from diversified branches of learning or practice, such that insights can be drawn upon and synthesized to influence a particular craft.

Anuj Magazine, an experienced tester from India has been working for quite sometime on Graphology and Software Testing.

I wanted aid his research and agreed to share my handwriting because his research is to identify the traits in successful testers. You can read the analysis of my handwriting by clicking on this link. I hope you'd enjoy reading the analysis and might be of help to Anuj if he approaches you.

I am sure there a lot of untold stories of what else other than testing has helped them to test better. I hope a lot of other testers come out with those stories that might benefit the community.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Automation replaces humans - The truth about what kind of humans it replaces

I often meet people who claim to be "automation testers" and I love and hate to argue with them about test automation. The more I meet such people, I am getting confident about making a conjecture that more than many people who claim to be doing test automating in India and other APAC country testers I visited, are doing it without knowing it or without wanting to know it.

Here are the things they say:

• Test automation replaces manual testing.
• Test automation is faster than manual testing.
• Test automation is cheaper than manual testing.
• Test automation is more reliable than manual testing.
• Test automation removes the monotonous work of a manual tester.
• Test automation provides confidence that the product works.
  
• Test automation is the future.
• Test automation is better than manual testing as a career option.
• Test automation tools provides Return On Investment.
• Test automation can find a lot more bugs than a manual tester.
• Test automation provides more job opportunities.
• Test automation is a way to go about testing.
• Test automation is about converting manual test cases to automation.

Lets look at some of them based on the conversation I have had with such people saying one or more of the above or the questions I'd like to ask you:

Test automation replaces manual testing and Test automation is the future

There are two planes standing there. One of them was tested by a trillion scripts that flew the plane for 400 hours and the other was tested by those 4 humans, flying 100 hours on it. If you were to board one of them, which one would you board and why?

None of those great human beings who said, "Test automation replaces manual testing" said, "I shall board the plane which was tested by a trillion scripts" and instead said, "I will take the plane that humans tested because I feel safer throughout the flight"

So did test automation replace manual testing even for those automating tests?
Test automation is the future of what?

Test automation is faster than manual testing & Test automation is cheaper than manual testing

Here are some statements I make. You can pick ones that makes most sense to you:

A. My Toshiba laptop runs faster than a McLaren Mercedes Formula1 car.
B. My Toshiba laptop runs faster than my old Acer laptop.
C. My Toshiba laptop runs faster on an XP than on Vista.
D. My Toshiba laptop runs faster than that horse which has a broken leg.
E. When someone needs help and I need to run to them, I use a test automation tool because that's faster.
F. QTP license is cheaper than hiring a 5 year experienced skilled tester.
G. QTP is expensive than Winrunner.
I. QTP is cheaper than a Ferrari engine.
J. Winrunner is cheaper than a business class ticket from Bangalore-Toronto.

Option C and G makes most sense. Why? Why not A , D, E, F, G, I, and J?

"Well, option C and G makes most sense to me because you are comparing the same Toshiba laptop when Vista and XP runs on it AND you are comparing two testing tools in G"

Is test automation [ using a software or a computer ] faster than humans to those who say that?
Is test automation cheaper than those human brains who claim to create it?

Test Automation is more reliable than manual testing

I first heard this eye opener, insightful idea from Michael Bolton , "People write code to find bugs on another code. They write code that is buggy which claims to find bugs on another code" and "Writing test scripts is another development project . Most of them don't realize that they are running two development projects in parallel and hence their main development project suffers."

So is test automation reliable than manual testing to those who write code that test another code?
We humans are fallible and all things we create are likely to be fallible, too. Test automation scripts are not excused by God to feel "All test scripts don't have bugs".

That doesn't mean human testing is more reliable. It means, anything done by humans is fallible. When you depend on a fallible thing that a fallible human created, you might not want to talk about reliability there.

Test automation removes the monotonous work of a manual tester

Why is testing monotonous? The unspoken truth!

Test automation is better than manual testing as a career option

Well, people think by becoming an automation tester they aren't doing manual testing. Here is what I have discovered - There are more people who claim themselves as automation testers in India than those who claim to be manual testers. So, let me know which is not manual testing? Can manual be termed as anything that does involve humans?

Lack of skilled testers at least has helped me grow phenomenal in my career. I am not THE skilled but I claim to be one of them and there might be very few like me and not many in India.

Test automation can find a lot more bugs than a manual tester & Test automation provides confidence that the product works

Yes, test automation is very intelligent. If a script is programmed to catch a bug when a value exceeds 5, it can also notice that the value did not exceed 5 and pass the test but the screen went blue for a while. If after all scripts have run and there aren't any bugs found, the test automation suite will alter its values and run once again to catch bugs. As the scripts run over and over again, the scripts learn the product well and if the management needs to take a decision to ship the product they would come near the script and say,

"Oh magical script, I want to ship this product. If you think it's a bad idea to ship now, speak out else remain silent"and then they listen to the script and decide whether to ship or not.

Yes, we are dumb. We can see a numerical value not exceeding 5 and pass the test case but also we can't spot the screen becoming blue for a while and do not report that as an issue while running a test that we claim to pass.


Test automation is about converting manual test cases to automation

Oh wow! So, I can't write automation scripts without writing cases and executing it at least once by my own? And if I write, many might cry "Foul".

Well, test automation scripts are just another program as the program it tests. Did developers have something they did manually before they wrote the code that you test with your manually converted automation test scripts?

Test automation tools provides Return On Investment.

Ok, so here is the industry standard of returns on a tool: For every $500 invested on a tool, a good ROI is about the tool helping the team find 50 bugs.

So, a good ROI for $9000 tool = ( 9000 / 500 ) * 50 = 900 bugs

If that doesn't make sense to you then what is the Return on Investment for a tool?

Bugs? Customer appreciation? The number of testers who get fired? The cost savings of hiring 10 skilled testers v/s the cost of buying the tool? The value that the tool brings to the testing effort? Tons of documentation produced? The added cost of training? The added cost of support? The added confusion that prevails? The number of fake experienced testers who apply to the job of a toolsmith?

Test automation is better than manual testing as a career option

Test automation, at least in India, is more chaotic than the so called manual testing. Hardly any tester knows why they are automating things. You may join any forum where testers are active discussing tools and automation, you'd see questions that the help file of the tool has answers and you would discover replies to such questions that is sometimes horribly wrong and yet the person who asked the question says, "Thanks" and the chaos continues.

There is more chaos among so called automation testers than so called manual testers because the so called automation testers form a huge community than the so called manual testers. More people with lack of insightful ideas means more chaos.

Want to know some useful information about automation or want ideas to think more about it?

Cem Kaner prefers to call most of the automation activities that most testers do as Computer Assisted Testing and I couldn't find evidence to refute the idea. I think it is a great idea to call it "Computer Assisted Testing" and you would know why it is a great idea if you go through this article by Cem or the PDF version of it.

So, all of us irrespective of calling us manual or automation testers do Computer Assisted Testing.

I know of an article written by James Bach that made many people who claims a lot of things about test automation feel guilty and I am sure if you have reached this place reading this post of mine, you'd be interested at reading James Bach's Test Automation Snake Oil - the paper or the slides .

I also know of a blog post from James - Manual tests cannot be automated and Shrini's post on A Mystery called automated testing.

Here is why we do different things in testing such as writing test code, executing test ideas, investigating, diversify approaches, think of different techniques, use heuristics, use oracles, Explore... - because each of them add different kind of value and each of them helps in finding different kinds of information or helps us ask new questions AND NOT because something is better than the other.

There is lot of thinking, reading and writing, I and you require, to know about test automation or about testing. The lazy ones don't get to learn and continue to spoil our craft because they don't stop writing, just like me. [ It's intentional ]

I know I haven't answered a question that you might have in mind but that's intentional again.

Update: Here is Jonathan Kohl's awesome interview about the same topic. Thanks to Konstantin for pointing it out through a comment.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --