Heroes of software testing - Do you know about them and their work?

My initial plan was to board the flight to Toronto on July 11th from Mumbai to attend CAST 08 software testing conference. Sangeeta, a tester from Mumbai influenced me to change my plan and organized a lecture in Ness Technologies in Mumbai.

I liked the offer and accepted it, as I knew I would enjoy the experience. It's fun to meet new testers, talk to them and learn new things. I am in constant search for budding testers like Sharath Byregowda , Ajay Balamurugadas , Sathish Kumar Chinappa , Sangeeta, Sandesh , Girija, Bhargavi and Nattu ( who accompanied me to CAST 08 ) and those who are passionate and do things about seeing a better tester community like Mohan Panguluri, COO of Edista Testing ... I never know where I can find similar ones. One of the ways that has been of great help to me, to find such testers and thinkers is, while at lectures and my workshops. I offer whatever help I am capable of doing to expose them to good stuff and I do it when they ask me for it.

Here is an excerpt from the talk:

"We are in India, the land where cricket is considered close to being a religion. It has super heroes like Sachin Tendulkar and Dhoni who have inspired a lot of people in India to play better cricket. They could inspire because a lot of people spent their time watching them play great cricket at tough situations. As Indians, you might have seen a lot of people who enjoy playing cricket the way Sachin and Dhoni play. As testers, have you seen anyone doing testing that has inspired you to test better?"

Response: Silence

and then a question comes up from a corner, "Who did Sir Don Bradman see as an inspiration for he is the best known cricketer of centuries?" and all other testers burst to laughter thinking that I was cornered with that question.

I took a while for the audience to settle down and replied,
"I know who inspired Sir Don Bradman" and then a pause to add, "It was himself."
"How many of you are inspired by your own testing?"

Response: Silence

End of excerpt

Reading good stuff as a bad practice and reading bad stuff as the best practice

The above state is what most Indian testers to my knowledge are stuck in. This state resulted because most of testers don't have the habit of reading as though its a bad practice. Most of those who read, are the ones who search in Google and land up at bad sites ( that includes my blog at times ). Most of those who Google search are the ones who are not interested at better thought process but at ready made best practice answers.

There are good and great stuff from internet. I once searched for expert testers and found James Bach. That google search changed my life and the way I test. There were a lot of junk stuff written to attract such search results that I had to pass through before finding James Bach's website. I was sure that I needed help about thought process from an expert tester and not ready made solutions from time to time.

The road to Toronto

I had to face a lot of trouble to get to Toronto. My Visa was rejected ( at the first attempt ), I didn't have sufficient funds ( as per the VISA officials and my bank statement ) , I was denied permission to board the plane as my ticket was booked via United States and I didn't have a US Visa. The Canadian customs had a discussion that bothered me if they would let me in after I landed at Toronto and then my baggage went missing and finally landed as the last one on the belt. Amidst all this was my hope to meet the heroes in real space who have inspired me and to meet those whom I can draw more inspiration from. I believe in God and also believe that God takes a human form to help humans. A great human who lives in 61, Ashburnham in Toronto and the one who lived in my house ( my dad ) helped me and cleared traps so that I reached Toronto and was able to attend CAST 08.


Chasing dreams

Finally I got a chance to meet the heroes who have been inspiring me to test and think better. My super hero James Bach wasn't present at the conference and hence I couldn't meet him. I would be meeting him in November at a conference where we both are invited to speak on testing at a developers conference in Malmo Sweden.

By meeting my heroes, I learned some important things that I could not have learned, had I not met them. I got an insight into the way they live, some ways they think and they communicate. It was several dreams of several years that came true, all in one place - CAST 08.

Not all learning is fun.

If you have understood what I have written in this post so far, then my English writing skill has improved a lot since I wrote my first blog post. It pained a lot to know that I was writing terribly bad English but had I not learned it, I wouldn't have been able to make sense to you in this post.

One of the important things I learned in CAST is the cost of making a few mistakes and the cost of wasting someone's time. I did some mistakes a few months ago but experienced the cost of making those mistakes when I could not stand in front of those people whose time I had wasted. I felt too ashamed of myself but I know I would not feel ashamed for a life time as I am recovering from those mistakes.

Learning is fun as long as you don't learn about yourself and how bad you are in somethings that you want to be really good at. The more I learn that I am bad, I am inspired to work hard enough to get over it. Being a student of James Bach means I keep searching for the answer, "How do I know what I know?" and often I discover what I don't know when I meet and discuss with great minds like the ones I met in CAST 08. I make it a practice to learn what I don't know or what more I'd want to learn on what I want to be good at.


Not all pain is bad

I went all way from Bengaluru ( Bangalore ) to Toronto to know more about myself and how bad I am in things I want to be good at. It hurts but there is no short cut to learning. For those who might disagree that learning could hurt - as testers we provide information about the quality of the product to the management. If we find a lot of bugs, it might hurt the management when they know it especially when it upsets their shipping schedules. It hurts them because they have learned something about their own product. This pain is then converted to measures to not get into a similar situation again. That is what I think some great man said, "No pain, No gain". The things that I learned from my heroes at the conference and other discussions would be reflected in my blog for years to come, so unfortunately keep reading them.


Person dependent software projects and organization

India depended on Sachin Tendulkar for winning a lot of matches. Cricket ( a team sport ) became a person dependent one. Is it a problem for Sachin?

Not really. Actually it demanded him to be at a very good knock and he loved that challenge as it challenged its consistency. It could also be stated that - he faced tough situations and cleared balls beyond boundaries which is why he is admired by millions worldwide.

Similarly, if your organization doesn't want to make their testing a person dependent thing it is because they don't like you to be Sachin Tendulkar . They can't stop from you becoming a Sachin of software testing because it depends on your learning as a tester and not they giving a learning opportunity to you.

You will be interested to read what James Bach wrote about the need for super heroes for software projects long back.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Automation replaces humans - The truth about what kind of humans it replaces

I often meet people who claim to be "automation testers" and I love and hate to argue with them about test automation. The more I meet such people, I am getting confident about making a conjecture that more than many people who claim to be doing test automating in India and other APAC country testers I visited, are doing it without knowing it or without wanting to know it.

Here are the things they say:

• Test automation replaces manual testing.
• Test automation is faster than manual testing.
• Test automation is cheaper than manual testing.
• Test automation is more reliable than manual testing.
• Test automation removes the monotonous work of a manual tester.
• Test automation provides confidence that the product works.
  
• Test automation is the future.
• Test automation is better than manual testing as a career option.
• Test automation tools provides Return On Investment.
• Test automation can find a lot more bugs than a manual tester.
• Test automation provides more job opportunities.
• Test automation is a way to go about testing.
• Test automation is about converting manual test cases to automation.

Lets look at some of them based on the conversation I have had with such people saying one or more of the above or the questions I'd like to ask you:

Test automation replaces manual testing and Test automation is the future

There are two planes standing there. One of them was tested by a trillion scripts that flew the plane for 400 hours and the other was tested by those 4 humans, flying 100 hours on it. If you were to board one of them, which one would you board and why?

None of those great human beings who said, "Test automation replaces manual testing" said, "I shall board the plane which was tested by a trillion scripts" and instead said, "I will take the plane that humans tested because I feel safer throughout the flight"

So did test automation replace manual testing even for those automating tests?
Test automation is the future of what?

Test automation is faster than manual testing & Test automation is cheaper than manual testing

Here are some statements I make. You can pick ones that makes most sense to you:

A. My Toshiba laptop runs faster than a McLaren Mercedes Formula1 car.
B. My Toshiba laptop runs faster than my old Acer laptop.
C. My Toshiba laptop runs faster on an XP than on Vista.
D. My Toshiba laptop runs faster than that horse which has a broken leg.
E. When someone needs help and I need to run to them, I use a test automation tool because that's faster.
F. QTP license is cheaper than hiring a 5 year experienced skilled tester.
G. QTP is expensive than Winrunner.
I. QTP is cheaper than a Ferrari engine.
J. Winrunner is cheaper than a business class ticket from Bangalore-Toronto.

Option C and G makes most sense. Why? Why not A , D, E, F, G, I, and J?

"Well, option C and G makes most sense to me because you are comparing the same Toshiba laptop when Vista and XP runs on it AND you are comparing two testing tools in G"

Is test automation [ using a software or a computer ] faster than humans to those who say that?
Is test automation cheaper than those human brains who claim to create it?

Test Automation is more reliable than manual testing

I first heard this eye opener, insightful idea from Michael Bolton , "People write code to find bugs on another code. They write code that is buggy which claims to find bugs on another code" and "Writing test scripts is another development project . Most of them don't realize that they are running two development projects in parallel and hence their main development project suffers."

So is test automation reliable than manual testing to those who write code that test another code?
We humans are fallible and all things we create are likely to be fallible, too. Test automation scripts are not excused by God to feel "All test scripts don't have bugs".

That doesn't mean human testing is more reliable. It means, anything done by humans is fallible. When you depend on a fallible thing that a fallible human created, you might not want to talk about reliability there.

Test automation removes the monotonous work of a manual tester

Why is testing monotonous? The unspoken truth!

Test automation is better than manual testing as a career option

Well, people think by becoming an automation tester they aren't doing manual testing. Here is what I have discovered - There are more people who claim themselves as automation testers in India than those who claim to be manual testers. So, let me know which is not manual testing? Can manual be termed as anything that does involve humans?

Lack of skilled testers at least has helped me grow phenomenal in my career. I am not THE skilled but I claim to be one of them and there might be very few like me and not many in India.

Test automation can find a lot more bugs than a manual tester & Test automation provides confidence that the product works

Yes, test automation is very intelligent. If a script is programmed to catch a bug when a value exceeds 5, it can also notice that the value did not exceed 5 and pass the test but the screen went blue for a while. If after all scripts have run and there aren't any bugs found, the test automation suite will alter its values and run once again to catch bugs. As the scripts run over and over again, the scripts learn the product well and if the management needs to take a decision to ship the product they would come near the script and say,

"Oh magical script, I want to ship this product. If you think it's a bad idea to ship now, speak out else remain silent"and then they listen to the script and decide whether to ship or not.

Yes, we are dumb. We can see a numerical value not exceeding 5 and pass the test case but also we can't spot the screen becoming blue for a while and do not report that as an issue while running a test that we claim to pass.


Test automation is about converting manual test cases to automation

Oh wow! So, I can't write automation scripts without writing cases and executing it at least once by my own? And if I write, many might cry "Foul".

Well, test automation scripts are just another program as the program it tests. Did developers have something they did manually before they wrote the code that you test with your manually converted automation test scripts?

Test automation tools provides Return On Investment.

Ok, so here is the industry standard of returns on a tool: For every $500 invested on a tool, a good ROI is about the tool helping the team find 50 bugs.

So, a good ROI for $9000 tool = ( 9000 / 500 ) * 50 = 900 bugs

If that doesn't make sense to you then what is the Return on Investment for a tool?

Bugs? Customer appreciation? The number of testers who get fired? The cost savings of hiring 10 skilled testers v/s the cost of buying the tool? The value that the tool brings to the testing effort? Tons of documentation produced? The added cost of training? The added cost of support? The added confusion that prevails? The number of fake experienced testers who apply to the job of a toolsmith?

Test automation is better than manual testing as a career option

Test automation, at least in India, is more chaotic than the so called manual testing. Hardly any tester knows why they are automating things. You may join any forum where testers are active discussing tools and automation, you'd see questions that the help file of the tool has answers and you would discover replies to such questions that is sometimes horribly wrong and yet the person who asked the question says, "Thanks" and the chaos continues.

There is more chaos among so called automation testers than so called manual testers because the so called automation testers form a huge community than the so called manual testers. More people with lack of insightful ideas means more chaos.

Want to know some useful information about automation or want ideas to think more about it?

Cem Kaner prefers to call most of the automation activities that most testers do as Computer Assisted Testing and I couldn't find evidence to refute the idea. I think it is a great idea to call it "Computer Assisted Testing" and you would know why it is a great idea if you go through this article by Cem or the PDF version of it.

So, all of us irrespective of calling us manual or automation testers do Computer Assisted Testing.

I know of an article written by James Bach that made many people who claims a lot of things about test automation feel guilty and I am sure if you have reached this place reading this post of mine, you'd be interested at reading James Bach's Test Automation Snake Oil - the paper or the slides .

I also know of a blog post from James - Manual tests cannot be automated and Shrini's post on A Mystery called automated testing.

Here is why we do different things in testing such as writing test code, executing test ideas, investigating, diversify approaches, think of different techniques, use heuristics, use oracles, Explore... - because each of them add different kind of value and each of them helps in finding different kinds of information or helps us ask new questions AND NOT because something is better than the other.

There is lot of thinking, reading and writing, I and you require, to know about test automation or about testing. The lazy ones don't get to learn and continue to spoil our craft because they don't stop writing, just like me. [ It's intentional ]

I know I haven't answered a question that you might have in mind but that's intentional again.

Update: Here is Jonathan Kohl's awesome interview about the same topic. Thanks to Konstantin for pointing it out through a comment.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

How to show product quality? - (Uncensored Version)

Notes from spying test experts

This is a top secret post and be very careful with the information. I am not sure if I can expose my notes to my dear blog readers but I am taking a risk of doing it.

There are some secrets of test experts that not many know. I had a mission sometime back to spy about how test experts actually find many important problems, quickly.

Many people think ( that included me, too) that Test Experts do lot more tests and have more fantastic test ideas and that's why they find many important problems, quickly.

I was hoping that there is more information hiding than what is visible about test experts and spying helped to uncover those hidden secrets.

This post is all about the secrets that I found spying test experts and the secrets of how they find many bugs?

You can trust me about the information to an extent that I and many other testers have been practicing the secrets that has resulted to our success of finding many important problems, quickly.


They just do one test to find many bugs!

Now, I am sure you don't want to believe me because my finding says that they do just one test to find many bugs. I am sure you have seen testers finding one bug per tests that find a bug and that makes you not believe this information but... you must understand that the secrets are always unbelievable.

You can feel safe after knowing a fact that I too didn't trust the secret of one test finding many bugs but my spying mission was to find out more granular details about it.

Looking more carefully into it, you might discover that it is not a test that finds a bug but it is a human that finds a bug and a test plays a role in helping the human find it. So there is something with humans that is helping to find bugs and not completely with the tests.

When I collected this much of information over months of spying, I was invited to be a part of the round table conference of experts as I had posed myself as a test expert from India (I knew I wasn't a test expert and even today I am too conscious that I am not one such) but you know... I am a spy and had to pose that way.

Over the round table conference, test experts started discussion about ORACLES* and I didn't know their terminology and just took notes. That evening, when I returned to my hotel, I was trying to connect ORACLES and the spying mission I had in hand.

* Oracles - principle or mechanism by which we (humans) identify problems

I was perhaps lucky that James Bach and Michael Bolton were staying in the same hotel that I was put up and we caught up at the dinner table.

I had an idea that I hope it worked. I was hoping that James Bach is a kind of person who would spill off the secrets in a context to prove me wrong and help me learn the secrets if I talked nonsense about ORACLES, because his passion to the craft of software testing is something that I am trying to match. Michael Bolton, the Birbal of North American software testing was a person whom I had to handle very carefully as he is more likely to identify me as a spy. At the end, I assumed that I was able to outsmart Michael in not letting him know that I am spy.

Going back to the hotel room, I started writing my discoveries about the secrets of test experts and the granular details of how they find important problems,quickly.

Before I publish a report to end the spying mission, I was sure that I had to practice the details I collected. I did practice and was excited that those ideas really worked and I found more bugs by executing one test.

Why many testers might not be able to practice the secrets that I am unlocking here is because they have a fixed expected result that is derived from a specification document that the tester thinks is The Holy Bible for the project.

Specification Document - is one single oracle and hence you might find one bug per test you execute.

The important thing is experts define a bug as "anything that threatens the value of the product" (-- James Bach)

Here are a list of Oracles that experts use with every test they do:

Consistency with the Image: When I execute a test, I try comparing and contrasting it with the image that my company or stake holders have been projecting about the product or the company.

For instance, if my company has an image in the market as people who produce software whose performance is good, and I execute a test whose documented expected result is "This functionality should work", I ask question, "Well it works but didn't it take too long to perform the operation?" and then I find a bug and say, "Although this operation goes through the time it took to complete the operation exceeds beyond the image the market has about us"

Consistency with Similar Product(s): When I execute a test the functionality might work as expected but I ask a question, "Well it works but doesn't it seem to perform in a way that users who are used to similar products might expect it to happen?" and then I find an important problem, quickly.

Consistency within the Product: When I execute a test that appears to have passed, I ask a question, "Well it works but doesn't it work in a different way than other areas of the same product?" and then I find an important problem, quickly.

Consistency with Claims: Marketing personnel and my manager are making certain claims about the product. When I execute a test and the test appears to have passed, I ask a question, "Well it works but does the way it works go against the claim that they make?" and then I find a bug.

Consistency with Statutes: When I execute a test that appears to have passed, I ask a question, "Well it works but does the way it works go against a law that an organization or a certifying authority might have?" and then I find a bug.

Consistency with User's Expectation: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in a way that the users whom I am aware of might not want it to happen?" and then I find a bug.

Consistency with History: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in the way the previous releases of this product work?" and then I find a bug.

While writing this post, I got an e-mail from James and Michael that reads...

Dear Pradeep,

We knew that you were spying us for a long time. We also saw the way in which you took notes, asked questions and drew inferences from what you could capture and that impressed us to help you learn more about testing because those are some important skills that a tester needs to have, to add good value for the cost customers pay for. We consider to hire you and we recommend you to quit spying profession and take up testing where you might get more laurels for your skills. Here is a link to know more about more secrets : www.satisfice.com/rst.pdf

Good luck!

-- James Bach & Michael Bolton

I was excited and I did quit spying, to take up testing as a full time activity. I thought I shouldn't post this in my blog because I didn't want those secrets to be given away for free and saved a draft of this post. I am afraid if Blogger has a bug that publishes drafts that are not intended to be published.


-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton


Update : Michael Bolton, blessed me with a post on his blog for this article of mine. I was jumping in joy for at least half an hour reading that and I am pleased to share it with you: http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html