It's a "tester" who finds a bug, even with the robust Google Search

The image you might see above is a screen shot of what I saw after hitting the 12th page for the search results of "tester". I am not sure if you can reproduce that because I haven't investigated on it but I did plan to capture it to demonstrate:

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it."

Be it with the robust systems like Google Search Engine or with weak systems that we might be using, it is always a HUMAN who finds a bug. A lot of testers I know think of "test case" finding a bug.

There is a test case document that consists of 9856985956895869698569956985698459698 test cases and no tester executing it wouldn't find bugs by itself. There is a test case document with 3 documented tests and a tester takes the help of that to find bugs when he executes, observes the result and recognizes a bug.

A test case is an extension of a test idea. What matters to a tester is a test idea and not the test case. Skilled (exploratory testers ( humans ) use tons of ideas ( heuristics and oracles ) to find and recognize bugs. That's why they can find more bugs that matter than those running thousands of test cases over and over again.

Honestly, 99% of testers I have come across didn't say - "I read each test case each time I have to execute it after I have done it once. Also, I religiously follow what is written in the test case".

What happens when they deviate from the documented test case is, they are exploring and running different test. Maybe they don't like to call it that way because their management who pays wouldn't like to know that they are not executing the "test cases".

That's how customers are fooled by management saying "yes, we are running test cases" and yet benefited by the testing community by running more tests.

A test idea can be executed in hundreds of different ways. Check out the deep analysis made by James Bach and Michael Bolton on What Do Scripts Tell Us?


--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Automation replaces humans - The truth about what kind of humans it replaces

I often meet people who claim to be "automation testers" and I love and hate to argue with them about test automation. The more I meet such people, I am getting confident about making a conjecture that more than many people who claim to be doing test automating in India and other APAC country testers I visited, are doing it without knowing it or without wanting to know it.

Here are the things they say:

• Test automation replaces manual testing.
• Test automation is faster than manual testing.
• Test automation is cheaper than manual testing.
• Test automation is more reliable than manual testing.
• Test automation removes the monotonous work of a manual tester.
• Test automation provides confidence that the product works.
  
• Test automation is the future.
• Test automation is better than manual testing as a career option.
• Test automation tools provides Return On Investment.
• Test automation can find a lot more bugs than a manual tester.
• Test automation provides more job opportunities.
• Test automation is a way to go about testing.
• Test automation is about converting manual test cases to automation.

Lets look at some of them based on the conversation I have had with such people saying one or more of the above or the questions I'd like to ask you:

Test automation replaces manual testing and Test automation is the future

There are two planes standing there. One of them was tested by a trillion scripts that flew the plane for 400 hours and the other was tested by those 4 humans, flying 100 hours on it. If you were to board one of them, which one would you board and why?

None of those great human beings who said, "Test automation replaces manual testing" said, "I shall board the plane which was tested by a trillion scripts" and instead said, "I will take the plane that humans tested because I feel safer throughout the flight"

So did test automation replace manual testing even for those automating tests?
Test automation is the future of what?

Test automation is faster than manual testing & Test automation is cheaper than manual testing

Here are some statements I make. You can pick ones that makes most sense to you:

A. My Toshiba laptop runs faster than a McLaren Mercedes Formula1 car.
B. My Toshiba laptop runs faster than my old Acer laptop.
C. My Toshiba laptop runs faster on an XP than on Vista.
D. My Toshiba laptop runs faster than that horse which has a broken leg.
E. When someone needs help and I need to run to them, I use a test automation tool because that's faster.
F. QTP license is cheaper than hiring a 5 year experienced skilled tester.
G. QTP is expensive than Winrunner.
I. QTP is cheaper than a Ferrari engine.
J. Winrunner is cheaper than a business class ticket from Bangalore-Toronto.

Option C and G makes most sense. Why? Why not A , D, E, F, G, I, and J?

"Well, option C and G makes most sense to me because you are comparing the same Toshiba laptop when Vista and XP runs on it AND you are comparing two testing tools in G"

Is test automation [ using a software or a computer ] faster than humans to those who say that?
Is test automation cheaper than those human brains who claim to create it?

Test Automation is more reliable than manual testing

I first heard this eye opener, insightful idea from Michael Bolton , "People write code to find bugs on another code. They write code that is buggy which claims to find bugs on another code" and "Writing test scripts is another development project . Most of them don't realize that they are running two development projects in parallel and hence their main development project suffers."

So is test automation reliable than manual testing to those who write code that test another code?
We humans are fallible and all things we create are likely to be fallible, too. Test automation scripts are not excused by God to feel "All test scripts don't have bugs".

That doesn't mean human testing is more reliable. It means, anything done by humans is fallible. When you depend on a fallible thing that a fallible human created, you might not want to talk about reliability there.

Test automation removes the monotonous work of a manual tester

Why is testing monotonous? The unspoken truth!

Test automation is better than manual testing as a career option

Well, people think by becoming an automation tester they aren't doing manual testing. Here is what I have discovered - There are more people who claim themselves as automation testers in India than those who claim to be manual testers. So, let me know which is not manual testing? Can manual be termed as anything that does involve humans?

Lack of skilled testers at least has helped me grow phenomenal in my career. I am not THE skilled but I claim to be one of them and there might be very few like me and not many in India.

Test automation can find a lot more bugs than a manual tester & Test automation provides confidence that the product works

Yes, test automation is very intelligent. If a script is programmed to catch a bug when a value exceeds 5, it can also notice that the value did not exceed 5 and pass the test but the screen went blue for a while. If after all scripts have run and there aren't any bugs found, the test automation suite will alter its values and run once again to catch bugs. As the scripts run over and over again, the scripts learn the product well and if the management needs to take a decision to ship the product they would come near the script and say,

"Oh magical script, I want to ship this product. If you think it's a bad idea to ship now, speak out else remain silent"and then they listen to the script and decide whether to ship or not.

Yes, we are dumb. We can see a numerical value not exceeding 5 and pass the test case but also we can't spot the screen becoming blue for a while and do not report that as an issue while running a test that we claim to pass.


Test automation is about converting manual test cases to automation

Oh wow! So, I can't write automation scripts without writing cases and executing it at least once by my own? And if I write, many might cry "Foul".

Well, test automation scripts are just another program as the program it tests. Did developers have something they did manually before they wrote the code that you test with your manually converted automation test scripts?

Test automation tools provides Return On Investment.

Ok, so here is the industry standard of returns on a tool: For every $500 invested on a tool, a good ROI is about the tool helping the team find 50 bugs.

So, a good ROI for $9000 tool = ( 9000 / 500 ) * 50 = 900 bugs

If that doesn't make sense to you then what is the Return on Investment for a tool?

Bugs? Customer appreciation? The number of testers who get fired? The cost savings of hiring 10 skilled testers v/s the cost of buying the tool? The value that the tool brings to the testing effort? Tons of documentation produced? The added cost of training? The added cost of support? The added confusion that prevails? The number of fake experienced testers who apply to the job of a toolsmith?

Test automation is better than manual testing as a career option

Test automation, at least in India, is more chaotic than the so called manual testing. Hardly any tester knows why they are automating things. You may join any forum where testers are active discussing tools and automation, you'd see questions that the help file of the tool has answers and you would discover replies to such questions that is sometimes horribly wrong and yet the person who asked the question says, "Thanks" and the chaos continues.

There is more chaos among so called automation testers than so called manual testers because the so called automation testers form a huge community than the so called manual testers. More people with lack of insightful ideas means more chaos.

Want to know some useful information about automation or want ideas to think more about it?

Cem Kaner prefers to call most of the automation activities that most testers do as Computer Assisted Testing and I couldn't find evidence to refute the idea. I think it is a great idea to call it "Computer Assisted Testing" and you would know why it is a great idea if you go through this article by Cem or the PDF version of it.

So, all of us irrespective of calling us manual or automation testers do Computer Assisted Testing.

I know of an article written by James Bach that made many people who claims a lot of things about test automation feel guilty and I am sure if you have reached this place reading this post of mine, you'd be interested at reading James Bach's Test Automation Snake Oil - the paper or the slides .

I also know of a blog post from James - Manual tests cannot be automated and Shrini's post on A Mystery called automated testing.

Here is why we do different things in testing such as writing test code, executing test ideas, investigating, diversify approaches, think of different techniques, use heuristics, use oracles, Explore... - because each of them add different kind of value and each of them helps in finding different kinds of information or helps us ask new questions AND NOT because something is better than the other.

There is lot of thinking, reading and writing, I and you require, to know about test automation or about testing. The lazy ones don't get to learn and continue to spoil our craft because they don't stop writing, just like me. [ It's intentional ]

I know I haven't answered a question that you might have in mind but that's intentional again.

Update: Here is Jonathan Kohl's awesome interview about the same topic. Thanks to Konstantin for pointing it out through a comment.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Ants solve problems that testers struggle to!

Ants can go into corners of tables and chair, which we humans might not be able to.
Ants can travel to any country without a passport or Visa (if they manage to slip into someone's baggage).
Ants can live anywhere and yet not pay rent or property tax.
Ants can eat junk food forever and live happily.

Doesn't all of them sound so obvious?

So, here is something that might not be so obvious about ants... Little creatures that we commonly refer to as ants have solved a problem that many testers today struggle to.

Have you heard a tester say, "I haven't tested this before / this is too complex / I don't know how to start testing it".

The important thing to note is that we might have heard it from our own mouth or heart and sometimes from others, too.

Here are some experiments that I did with ants to see how they respond to complexity:

Experiment 1

I took an ounce of sugar, placed it near a colony of ants and observed what happened to the sugar I placed after a while.

My experiment resulted in each ant carrying a crystal of sugar and moved them to their nest. Sugar is all gone!

Experiment 2

I placed a cake of size approximately 1000 times bigger than an ant in the same place and observed what happened to the cake.

This experiment resulted in each ant breaking the huge cake into smaller fragments and moved it to their nest bit by bit. Cake is all gone!

Experiment 3

I put a piece of solidified sugar candy near the same colony, which was hard to break for an ant.

This experiment resulted in ants trying to break the candy to smaller segments, but probably the smart ants realized it to be complex than their earlier two assignments I gave them and then they all co-ordinated together and carried the candy to their nest. Candy is all gone!

I bet I could place a mammoth of food for them and they, without bothering about the complexity of it, would try moving it to their nest or create more nests and colonies to accomodate it or work a strategy to get it to their nest or much it as much as they can till the mammoth lies there...

Ben Simo, a wonderful tester you might already know, recently wrote a post titled Solving Intractable Problems and ended it beautifully by saying, "Start with what you recognize" - which I now think is what ants do instead of saying, "That food(work) is not for us, it is for bigger (skilled) ants"

After having done these experiments I was ashamed of my behavior of hesitating to test a product (which I did a couple of years ago) just because something appeared to be too complex or I thought I didn't know about the product or technology.

No, I said, *I* was ashamed and I am not going to ask if you were ashamed because you might have not done that.

After all they are ants and they don't have the super brains we have and lets ignore them folks and continue to live as happy as we are, complaining that things are complex.

Silly ants, they don't even know they are dealing with complexity. I think they should be ashamed of it.


Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

The story of a monk who sold bugs!

In January 2006, I wrote an interesting post in my blog that became one of the favorite e-mail forward among testers in India. To my disappointment, when I got the forward e-mail from a couple of people, I could see different names as authors in it and I never got the credit. I bet no one can make the kind of creative grammatical mistakes that I do and its always easy to spot my writing. Also as Michael Bolton pointed out in software-testing list, "Pradeep writes from heart".

Fearing those idiots who did not owe me credits I didn't want to write such stories that could make another interesting forward with a claim that they are the authors. I am no coward and most important of all, I don't want my blog readers miss such stories that I have in mind. So here is a story that I think that has gone weeks of thinking:

The story of a monk who sold bugs written by Pradeep Soundararajan [Idiots and fools can change it to their name if wanting to forward the story]

Once upon a time, not so long ago, a monk set out looking for a job in India that could help him earn enough to feed his family. Monk knocked the door of a few software companies and by God's grace, a company did consider him for an opening in testing that they had.

The monk was new to the software industry and had calculated his monthly salary by dividing the CTC ( or cost to company per annum ) by 12 and felt happy about it but when he received his first pay check, he did come to know that maths in software industry, especially when it comes to salary is different. It was not possible to run his family with the monthly pay check he received and hence had to find a way to earn more.

Looking out for another job means, taking time off the work, which also implies that he earns lesser for that month and might end up not getting an offer. So what is the way to work in the same company and yet earn more?

In conversation with other testers in the company during lunch, the monk heard a couple of testers lament about the development teams not fixing the bugs found by them. One tester said, "I bet a 100 rupees on each of these bugs" and other testers too started making similar statements. The monk who was listening to this got an idea and asked all these testers, "did you say 100 rupees to convince developers to fix a bug?" and then testers replied, "Oh yes!"

The monk went back to the monastry in search of his Guru to seek help to win the bet that was placed on each bug. The Guru had left for a world tour but had left a note to all the monks who came in search of him and the note said, "Your problems will be solved even if I am not here to help you and the one who can help you this time is yourself". After reading this the monk didn't get disappointed but thanked his Guru for leaving such a wonderful note.

The monk boarded a bus to head back to office and at the bus he handed over a 10 rupee note to purchase a ticket. The conductor said, "Sorry, I cant give you the ticket, the low battery condition in this ticket generator will make the ticket get stuck during the print out and its a very painful job to remove the paper out and fix the roll in the position and hence we are losing money and business because of this" and the monk said, "Thank you! I got the answer for a question that was bothering me".

The monk, on reaching office called for a meet with other testers and asked them to bring their bug report print outs. Everyone gave their bunch of reports and the monk started his homework.

He picked a bug that was logged with a low battery operation of an embedded product where the developers had commented "Users are not supposed to use the product in low battery". The monk went to the developer and narrated the incident that happened in bus and also added that, "I guess the government is planning to sue the company that provided that machine. As ours is a huge company where different products are developed, are you sure its not our company thats going to be sued?"

That question to a developer was enough for the monk to make his first 100 rupees. As days passed, the monk narrated a lot of real time stories to developers who had offered reluctance to fixing a lot of bugs assigned to them and the monk made a lot of money. As the stories from the monk became popular among the developers in the company, developers loved listening to monk' stories and the monk had a very high credibility among all teams.

The testers conducted a meet and discussed about the monk who made a lot of money through them and decided to cancel their bets thereafter and some testers said, "Ah! I can say better stories, what big deal? Let me convince the developers hereafter and I am not going to pay that already rich monk anymore"

The monk had not only caused a revolution within himself, he had also created an evolution, the way other testers wanted to work.

The monk decided to go to a new place looking for more and different kind of problems. He made a lot of money solving such problems and eventually became one of the richest tester of the world!

Lessons to learn from the story of a monk who sold bugs:

• Everyone needs something convincing to take up a task, be it fixing a bug or executing scripts.
• Developers need some real time scenarios, facts, customer information and information about customers knowledge, models in which the customer uses a product in a packaged form - story - to get convinced to fix bugs.
• One who constantly thinks of solving a problem, will find the solution anywhere.
• The most reliable person for help, is yourself.
• Testers need to develop and practice a skill of story telling to get better at selling bugs.
• Testing is not an activity of improving quality, its an activity of finding information and it is the product/development teams who fix those issues for improving the quality. In case you still want to think that testing is an activity of improving the quality, you might want to become a best bug seller to continue thinking of it.
• Testing is not necessarily a computer science subject and its life science, social science, ... all science and all non science, too.
• There is a traditional way of making money and as a tester if you take the traditional way, you make less with it.
• Some bugs are born with self explanatory story but some aren't and testers need to build them.
• More your sales figure ( selling bugs, I mean) higher is your credibility and as testers if you don't have credibility, you might not be able to boost your sales figures further.
• Pradeep Soundararajan will keep writing such stories without fearing the people who try putting their name as authors and exposing themselves as idiots, fool.

-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton