"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Tuesday, March 18, 2008

Automation replaces humans - The truth about what kind of humans it replaces

I often meet people who claim to be "automation testers" and I love and hate to argue with them about test automation. The more I meet such people, I am getting confident about making a conjecture that more than many people who claim to be doing test automating in India and other APAC country testers I visited, are doing it without knowing it or without wanting to know it.

Here are the things they say:
  • Test automation replaces manual testing.
  • Test automation is faster than manual testing.
  • Test automation is cheaper than manual testing.
  • Test automation is more reliable than manual testing.
  • Test automation removes the monotonous work of a manual tester.
  • Test automation provides confidence that the product works.
  • Test automation is the future.
  • Test automation is better than manual testing as a career option.
  • Test automation tools provides Return On Investment.
  • Test automation can find a lot more bugs than a manual tester.
  • Test automation provides more job opportunities.
  • Test automation is a way to go about testing.
  • Test automation is about converting manual test cases to automation.
Lets look at some of them based on the conversation I have had with such people saying one or more of the above or the questions I'd like to ask you:

Test automation replaces manual testing and
Test automation is the future

There are two planes standing there. One of them was tested by a trillion scripts that flew the plane for 400 hours and the other was tested by those 4 humans, flying 100 hours on it. If you were to board one of them, which one would you board and why?

None of those great human beings who said, "Test automation replaces manual testing" said, "I shall board the plane which was tested by a trillion scripts" and instead said, "I will take the plane that humans tested because I feel safer throughout the flight"

So did test automation replace manual testing even for those automating tests?
Test automation is the future of what?

Test automation is faster than manual testing & Test automation is cheaper than manual testing

Here are some statements I make. You can pick ones that makes most sense to you:


A. My Toshiba laptop runs faster than a McLaren Mercedes Formula1 car.

B. My Toshiba laptop runs faster than my old Acer laptop.
C. My Toshiba laptop runs faster on an XP than on Vista.
D. My Toshiba laptop runs faster than that horse which has a broken leg.
E. When someone needs help and I need to run to them, I use a test automation tool because that's faster.
F. QTP license is cheaper than hiring a 5 year experienced skilled tester.
G. QTP is expensive than Winrunner.
I. QTP is cheaper than a Ferrari engine.
J. Winrunner is cheaper than a business class ticket from Bangalore-Toronto.

Option C and G makes most sense. Why? Why not A , D, E, F, G, I, and J?

"Well, option C and G makes most sense to me because you are comparing the same Toshiba laptop when Vista and XP runs on it AND you are comparing two testing tools in G"

Is test automation [ using a software or a computer ] faster than humans to those who say that?
Is test automation cheaper than those human brains who claim to create it?

Test Automation is more reliable than manual testing

I first heard this eye opener, insightful idea from Michael Bolton , "People write code to find bugs on another code. They write code that is buggy which claims to find bugs on another code" and "Writing test scripts is another development project . Most of them don't realize that they are running two development projects in parallel and hence their main development project suffers."

So is test automation reliable than manual testing to those who write code that test another code?
We humans are fallible and all things we create are likely to be fallible, too. Test automation scripts are not excused by God to feel "All test scripts don't have bugs".

That doesn't mean human testing is more reliable. It means, anything done by humans is fallible. When you depend on a fallible thing that a fallible human created, you might not want to talk about reliability there.

Test automation removes the monotonous work of a manual tester

Why is testing monotonous? The unspoken truth!


Test automation is better than manual testing as a career option

Well, people think by becoming an automation tester they aren't doing manual testing. Here is what I have discovered - There are more people who claim themselves as automation testers in India than those who claim to be manual testers. So, let me know which is not manual testing? Can manual be termed as anything that does involve humans?

Lack of skilled testers at least has helped me grow phenomenal in my career. I am not THE skilled but I claim to be one of them and there might be very few like me and not many in India.

Test automation can find a lot more bugs than a manual tester & Test automation provides confidence that the product works

Yes, test automation is very intelligent. If a script is programmed to catch a bug when a value exceeds 5, it can also notice that the value did not exceed 5 and pass the test but the screen went blue for a while. If after all scripts have run and there aren't any bugs found, the test automation suite will alter its values and run once again to catch bugs. As the scripts run over and over again, the scripts learn the product well and if the management needs to take a decision to ship the product they would come near the script and say,

"Oh magical script, I want to ship this product. If you think it's a bad idea to ship now, speak out else remain silent"and then they listen to the script and decide whether to ship or not.

Yes, we are dumb. We can see a numerical value not exceeding 5 and pass the test case but also we can't spot the screen becoming blue for a while and do not report that as an issue while running a test that we claim to pass.


Test automation is about converting manual test cases to automation

Oh wow! So, I can't write automation scripts without writing cases and executing it at least once by my own? And if I write, many might cry "Foul".

Well, test automation scripts are just another program as the program it tests. Did developers have something they did manually before they wrote the code that you test with your manually converted automation test scripts?

Test automation tools provides Return On Investment.

Ok, so here is the industry standard of returns on a tool: For every $500 invested on a tool, a good ROI is about the tool helping the team find 50 bugs.

So, a good ROI for $9000 tool = ( 9000 / 500 ) * 50 = 900 bugs

If that doesn't make sense to you then what is the Return on Investment for a tool?

Bugs? Customer appreciation? The number of testers who get fired? The cost savings of hiring 10 skilled testers v/s the cost of buying the tool? The value that the tool brings to the testing effort? Tons of documentation produced? The added cost of training? The added cost of support? The added confusion that prevails? The number of fake experienced testers who apply to the job of a toolsmith?

Test automation is better than manual testing as a career option

Test automation, at least in India, is more chaotic than the so called manual testing. Hardly any tester knows why they are automating things. You may join any forum where testers are active discussing tools and automation, you'd see questions that the help file of the tool has answers and you would discover replies to such questions that is sometimes horribly wrong and yet the person who asked the question says, "Thanks" and the chaos continues.

There is more chaos among so called automation testers than so called manual testers because the so called automation testers form a huge community than the so called manual testers. More people with lack of insightful ideas means more chaos.

Want to know some useful information about automation or want ideas to think more about it?

Cem Kaner prefers to call most of the automation activities that most testers do as Computer Assisted Testing and I couldn't find evidence to refute the idea. I think it is a great idea to call it "Computer Assisted Testing" and you would know why it is a great idea if you go through this article by Cem or the PDF version of it.

So, all of us irrespective of calling us manual or automation testers do Computer Assisted Testing.

I know of an article written by James Bach that made many people who claims a lot of things about test automation feel guilty and I am sure if you have reached this place reading this post of mine, you'd be interested at reading James Bach's Test Automation Snake Oil - the paper or the slides .

I also know of a blog post from James - Manual tests cannot be automated and Shrini's post on A Mystery called automated testing.

Here is why we do different things in testing such as writing test code, executing test ideas, investigating, diversify approaches, think of different techniques, use heuristics, use oracles, Explore... - because each of them add different kind of value and each of them helps in finding different kinds of information or helps us ask new questions AND NOT because something is better than the other.

There is lot of thinking, reading and writing, I and you require, to know about test automation or about testing. The lazy ones don't get to learn and continue to spoil our craft because they don't stop writing, just like me. [ It's intentional ]

I know I haven't answered a question that you might have in mind but that's intentional again.

Update: Here is Jonathan Kohl's awesome interview about the same topic. Thanks to Konstantin for pointing it out through a comment.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

26 comments:

Anonymous said...

Hi,

You rocked again Pradeep :-)

Venkat Reddy Chintalapudi said...

@ Pradeep,

Yet another nice post & the perceptions do differ on the jargon being used here.

But, the tools do help humans to bring value addition in their set of activities.

In the current gloabl villeage, it's tough to think of complete manual / automation.

So the same applies to Testing too, the tools does help to bring more value addition in our set of activities. But the perceptions & usage differs from person to person.

Cheers,
Venkat.

Pradeep Soundararajan said...

@Venkat,


But, the tools do help humans to bring value addition in their set of activities.


Yes, that's one of the point this post re-iterated. But it is not just any activity a tester does. Watch for the following comment...


So the same applies to Testing too, the tools does help to bring more value addition in our set of activities. But the perceptions & usage differs from person to person.


The idea of what it is might differ from person to person but the truth remains same - tools are extended hands for a tester to accomplish his mission.

Thanks for commenting.

Ken De Souza said...

I like the points you have to make regarding how test automation is typically approached. I have experienced this and now I'm in the process of educating management on how "automation" is an overloaded term. In fact, I prefer Corey Goldberg thought on the term "programmatic testing" instead of "computer assisted testing". The term is somewhat overloaded.

Great post though!

Pradeep Soundararajan said...

@Ken de souza,

"Computer Assisted Testing" makes great sense to me for everything we call as test automation.

It is good to know that you felt the term is overloaded. This brings a good example for what one thinks "overloaded" could differ from another and so on in the process of educating your management you might want to try using both terms and let them judge what is overloaded to them and what makes great sense.

Thanks for the comment. I enjoyed replying to your comment.

eddie said...

Okay while I agree with what you say in userspace application.. when it comes down to kernel testing it becomes a little bit harder.. don't you think?

Kernels provide you with system calls.. System calls are mainly used in applications you right.. Wouldn't in this case, automation be the best choice?

Pradeep Soundararajan said...

@Eddie,

It is common that when people read anything against automation they come up with contexts and ideas to challenge. However this post is *NOT* about manual testing v/s automated testing and a proof of manual testing is better than the latter.

Okay while I agree with what you say in userspace application.. when it comes down to kernel testing it becomes a little bit harder.. don't you think?

If automation is humans writing scripts to test a product and the product under test is a kernel then writing code to test it makes most sense but however that's Computer Assisted Testing with human involvement in it.

Kernels provide you with system calls.. System calls are mainly used in applications you right.. Wouldn't in this case, automation be the best choice?

Here is the universal best choice for any given problem: Using brains!

When you say "best" what are you trying to compare it with?

Skilled testers do what the context demands to do and if the context demands writing test scripts, they do it.

Thanks for asking questions. You might have helped someone who had a similar question to get it cleared.

eddie said...

Skilled testers do what the context demands to do and if the context demands writing test scripts, they do it.


Well, since, hypothetically, the kernel provides a definite interface. Testing that interface via automation and reused scripts would make it cheaper, faster and most likely reliable.. right? Well obviously you need some hu(wo)man, who know what they are doing, to add to the automated suite. Or maybe I just got those two terms [automate/manual] totally wrong again.

You have an interesting blog, I just discovered it. Am not challenging or being against manual testing not at all. I just wanted to see your views on automation in kernel testing, and the benefit of automation for that manner over manual testing.

Pradeep Soundararajan said...

@Eddie,

Thanks for the reply and I value your time.

Am not challenging or being against manual testing not at all. I just wanted to see your views on automation in kernel testing, and the benefit of automation for that manner over manual testing

I understand but I use a word "challenge" to all questions that help me bring out more information that helps people in communication things.

Testing that interface via automation and reused scripts would make it cheaper, faster and most likely reliable.. right? Well obviously you need some hu(wo)man, who know what they are doing, to add to the automated suite.

Automation or lets start calling it Computer Assisted Testing is an approach to test software that adds value in finding information about the quality of the product under test.

If we have an approach that is valuable in a specific context, it is a good idea to do that but as you said it is important to have humans who know what they are doing.

Test automation is expensive at least because skilled people working on it are expensive.

When we ( humans ) say cheaper, faster, easier... what are we comparing it with?

If you are comparing a script written by me and a script written by you and then say, "My script runs faster than your's", I'd appreciate that.

Please help me to appreciate how I'd compare a script with a human being in test execution.

Anonymous said...

Hi Pradeep,

Instead of all these discussions, debates, analysis..........
Why cant we coin a term which refers a tester as both manual + automated and generalize all testers under it.
Because ( as per my view )
1. All manual tester also uses tools and hence cannot be termed as a manual tester
and
2. All automated tester has to do manual testing before writing scripts atleast @ some point of time.

Any comments?????

Regards,
Tester

Pradeep Soundararajan said...

@Anonymous Tester,

Why cant we coin a term which refers a tester as both manual + automated and generalize all testers under it.

I am not a fan of coining new terms in testing when there exists a more meaningful one already available.

Dr Cem Kaner has suggested Computer Assisted Testing and that makes great sense even to the ideas you have shared.

Shrini Kulkarni said...

Good post pradeep -- in your usual unique style ...

I have few comments to make ..

>>> Test automation is more reliable than manual testing.

Pradeep, Here is a situation where I can make above statement. When it comes automation results, I can be sure that they are correct (assuming that autmation code is correct) - that is a notin of reliability. It could be different matter that those correct(reliable) results may not make any sense - the information provided by those results may be meaning less. For some, say in regulated environments like Pharma - it is the reliability of results matters than anything else.
If you are just interested in knowing result and their reliability and nothing else -- automation can REALLY HELP. I agree humans are unreliable on THIS aspect (not by intention but by inherrent human fallibility)


>>>Test automation provides confidence that the product works.

Interestingly enough, this is not heard that much - not a popular belief. People belive that Automation helps to cut down test cycle time. Luckily, in my exp, people do not talk about "confidence in quality" in the same breath as "a tool to reduce the cycle time" - that I think is a GOOD sign.



>>>Test automation is the future.
This is another thing - I have not heard it that much.


>>Test automation is better than manual testing as a career option.

This is true to larger extent in india. I will tell you why. Most managers in india - do not understand clearly the skills required for sapient testing. Most of them confuse testing skills to domain skills (banking, insurance etc) or process skills ( CMM, six sigma etc). With so many bad examples around - they typically resort to check testers on something that they are sure of. That is how automation tools take center stage in hiring of testers at entry and middle level.

>>Test automation tools provides Return On Investment.
I am not sure about your opinion here. Are you saying automation tools provide ROI on Testing? Let me tell you RoI from automation is thought independently of testing ROI (thank GOD .. it is seperate worry/business case to make).

This is how people make case for/ROI model for automation. Current manual testing cycle (what ever is approach) takes 5 days. Let us automate 50% of the test cases and that would result in cycle time reduction of 50%. In some cases, where ROI is demonstrated by tool vendors - it is using this model.


>> Test automation can find a lot more bugs than a manual tester.

You know what - I have seen, people dont expect to find bugs by automation. A 100% pass result from automation is the most pleasing thing for a test manager.
What people value most in automation is faster execution regardless of what.

>>>Test automation provides more job opportunities.
Here is a rephrase ... Test automation engineer as job title is more lucrative than Test engineer. As indicated already - if you know an automation tool (no testing knowledge) you are more employable than otherwise. This is sad but true.

only few test managers today in india know that A good automation engineer is a good tester first than anything else.

>>>There are two planes standing there. One of them was tested by a trillion scripts that flew the plane for 400 hours and the other was tested by those 4 humans, flying 100 hours on it. If you were to board one of them, which one would you board and why?

Pradeep - if someone asks this question - I would refuse them to answer and say "insuffient information to make a decision".
You seem to be showing a decision in favor of that plane (aero-plane) as good one. The whole example of human testing vs machine testing in this case is not conviencing one. James Bach mentions - automation is a different type of testing. In a typical context of testing (human), remove the human from the scene - the whole thing changes.
If I think of a test, as human, I have an option to execute or make another program to test. So human execution and programmatic execution are two different and uncomparable means of doing testing. I might decide to fly a plane tested by an automation program provided I have other information to confirm my decision.

It is good thing to notice that people are not talking NOW to replace human tester --- yes it is trend a welcome trend. What people are now saying x % of automation would result in x% in reduction of test effort. Hence those testers can be used in other projects. I agree this is also a type of replacement but people talking about firing testers on the basis of automation has come down. Thank god.

>>> When you depend on a fallible thing that a fallible human created, you might not want to talk about reliability there.

That is a great point. I like the way you have said it. I would like to make an important point about human fallibility. It is often mistaken with "intentional" and careless behavior. Fallibilty of any human (assuming that intention is good) is more to do with inherrent shortcoming of individuals involved and one can work at improving such shortcomings. Here is a heuristic -"Everything in software is fallible since it is created by human" Can you think of 3 ways in which this heuristica can fail?

>>> Yes, we are dumb. We can see a numerical value not exceeding 5 and pass the test case but also we can't spot the screen becoming blue for a while and do not report that as an issue while running a test that we claim to pass.

Good point again ...Automation can spot discrepancy (not anything else) according to a programmed condition. In other wards, checking mechanism is static and pre-programmed. But software under test constantly under goes modification. A skilled human can adjust to this - automation program can not. That is the difference. Yes, there is a way you don't want the checking program or mechanism NOT to change as you would like to check the differences or consistency. There automation can prove to be helpful it is untiring, 24x7 and accurate. As manager we should clearly check under what circumstances we need such types of results.

>> Test automation is about converting manual test cases to automation

This is an interesting one.

According to Cem Kaner (high volume test automation - paper) A typical automation paradigm involves a human tester writing a test cases first, executing that, noting the results and introducing automation. I sense that automation requires test cases be documtated in some form before they could be picked up for automation. I am talking about some kind of "written down" test cases as pre requisite to automation. In some sense, an automated test is a programm equivalent of a manual test case.

James Bach seem to take a different view on this. James feels that it may not be necessarily think about manual test case design and documentation and Automation as sequential activities in that order.

A tester can think about test (automation can not *think* about a test) and tester can then use a program to run the full or part of the test. So test is born in the mind of a tester who subsequently decide to execute part or full of the test all himself or by another program.

If I use a program to generate a set of test cases which a human can execute - what would you call such automation as? Ben Simo can fill you with Model based test case generation fundaas ..

>>> There is more chaos among so called automation testers than so called manual testers because ..

Make no mistake .. there are no automation testers or automated testing. By definition any good testing is human not by machines. you can not automate testing (hence there is nothing called automated testing exists). You can automate portions and phases of testing.

I prefer the word computer assisted testing (Ben Simo, I think, might say - even a pure manual testing is computer assisted testing as you use computer to do *any* kind of software testing (so a computer is always involved - unless you do other types of testing that does not involve computers at all)

Matt Heusser and I exchanged lots of automation related topics .. check out his blog for references.


BTW, this post of your has lots in common with my 10 classic reasons why test automation projects fails - STAR EAST 2007 presentation. You might want to host that ppt and link to this post

Shrini

Pradeep Soundararajan said...

Pradeep, Here is a situation where I can make above statement. When it comes automation results, I can be sure that they are correct (assuming that autmation code is correct) - that is a notin of reliability.

You mean if a code is wrong, it performs wrong each time it is executed and is reliably wrong.

It could be different matter that those correct(reliable) results may not make any sense - the information provided by those results may be meaning less. For some, say in regulated environments like Pharma - it is the reliability of results matters than anything else.

I think I’d like to highlight the discussion going on in CDT or software-testing@yahoogroups.com on how rounding off 15th decimel points have affected business and the classic Excel example by professor Kahan: http://www.cs.berkeley.edu/~wkahan/ARITH_17.pdf

<< If you are just interested in knowing result and their reliability and nothing else -- automation can REALLY HELP. I agree humans are unreliable on THIS aspect (not by intention but by inherrent human fallibility) >>

If humans are unreliable how can the code they write be reliable. Are you saying computers are reliable? The electricity that helps running a computer reliable? The processor is reliable?

<< I am not sure about your opinion here. Are you saying automation tools provide ROI on Testing? Let me tell you RoI from automation is thought independently of testing ROI (thank GOD .. it is seperate worry/business case to make). >>

Are you saying a Testing tool ROI has no impact on Testing ROI although people might not link it?

<<
This is how people make case for/ROI model for automation. Current manual testing cycle (what ever is approach) takes 5 days. Let us automate 50% of the test cases and that would result in cycle time reduction of 50%. In some cases, where ROI is demonstrated by tool vendors - it is using this model. >>

Yeah and those people are fools, in my opinion.

<< You know what - I have seen, people dont expect to find bugs by automation. A 100% pass result from automation is the most pleasing thing for a test manager.
What people value most in automation is faster execution regardless of what. >>

Another set of people whom I’d like to call them fools.

>>>There are two planes standing there. One of them was tested by a trillion scripts that flew the plane for 400 hours and the other was tested by those 4 humans, flying 100 hours on it. If you were to board one of them, which one would you board and why?

Pradeep - if someone asks this question - I would refuse them to answer and say "insuffient information to make a decision".

Well I think me and you have boarded the plane without knowing this information by trusting the airline service provider.

Any amount of information might be insufficient but all of us take decisions based on the available information although we’d loved to have more.

BTW, this post of your has lots in common with my 10 classic reasons why test automation projects fails - STAR EAST 2007 presentation. You might want to host that ppt and link to this post

I did link to your blog post because I felt like giving that information to my readers and I’d like to link your presentation if I blog such a thing.

Thanks!

Konstantin said...

Hello,

I should say a very interesting thoughts and a great discussion above. I enjoyed reading it all!

Probably you guys have seen that, but there was a nice interview with Jonathan Kohl regarding the automation testing (or "Computer Assisted Testing" as you prefer ;)) a month ago. A must read as well!

I just want to comment on this one:

>>Test automation is better than manual testing as a career option.

For good or bad this is true all around the world I guess. And it makes sense since managers rely on your experience with the testing scripts automation -- in most cases if you've worked with automated scripts, you can do it again, correct? ;) Otherwise you have to start of the automation thinking from the beginning when they hire you.

The manual testing skills are still hardly to examine in through single task or just an interview. Based on my experience, the vision of a tester and of course her desire to explore is what one should rely on.

Thanks for the nice reading, Pradeep!

-Konstantin

Pradeep Soundararajan said...

@Konstatin,

Thanks for giving the link to John Kohl's interview. That's awesome stuff.

Many thanks for your comment, too.

PP said...

Hola Pradeep,

When i read this article i felt ...oh! this is the same stuff , i know this..its commonsense that automation testing is not answer to all the testing we require and why we are thinking or considering questions like "Will it replace Manual testing".

Then i discuss something related to this in my office (currently in spain so onsite-office)
and i feel yes there are still people who are awe-struck by automated testing..i thought the term automated testing should have got outdated by now..but they are still behaving as it is something that is going to change the testing world..I am not sure may be i am reading too much James bach,Michael ..Shrini and ofcourse you and thinking world has understood this concept...:)

¡Hasta luego!

Pradeep Soundararajan said...

@PP,

When i read this article i felt ...oh! this is the same stuff , i know this..its commonsense that automation testing is not answer to all the testing we require and why we are thinking or considering questions like "Will it replace Manual testing".

I don't know who gave a name "common sense" but I think such a thing does not exist for sure.


Then i discuss something related to this in my office (currently in spain so onsite-office)
and i feel yes there are still people who are awe-struck by automated testing..i thought the term automated testing should have got outdated by now..but they are still behaving as it is something that is going to change the testing world.


It might certainly change the world when they try to think deep or when they try to unlearn what they think "automated testing" is and learn the term "Computer Assisted Testing".

I am not sure may be i am reading too much James bach,Michael ..Shrini and ofcourse you and thinking world has understood this concept...:)

Well, I am not an expert. I am a human being. I am fallible. The ideas I share in this blog are not excused of fallibility. I am still trying to learn testing.

Have a nice time in Spain and send me some snaps of Spain, I'd love to visit Spain.

Anonymous said...

Hi,

This is not up to the mark pradeep.

Bye

Pradeep Soundararajan said...

@Anonymous,

This is not up to the mark pradeep.

This is not up to whose mark?
This is not up to what mark?
Why should this be up to an unknown mark?

What do you mean by "this"?

Anonymous said...

Hey Pradeep,

Here is a story I would like to relate to you just to look at the bright side of automation! I know you aren't against automating per se, but you dislike folks who swear by it!

I had joined this interesting project that basically involved viewing & querying OLAP data on MS Spreadsheets. We had about 300 tests in regression and new tests added for upcoming features. We had dozens of configurations to test against, different spreadsheet versions, different database versions, mid-tier versions etc. Though a relatively small project in the whole BI stack, however running these certification tests for new BI releases had become a nightmare! Sometimes, we didn't have any changes on our side, but since there were dependent library changes we had to regress! Add to this the fact that upper management always wanted us to execute long regressions at the drop of a hat! Now, I don't think, but I know that executing 300 tests again & again is donkey work! Though I don't mean to undermine the importance of these regressions! I knew that our set of 300 regression tests was less than perfect and there were many more bugs lurking around the corners that needed a little deeper digging. I don't mean the "text box a few pixels to the right" kind of bugs, but the real bugs that real customers care about!
So we tried using WinRunner, QTP, Silk etc to get the 300 tests off our backs, but nothing seemed to work well! Why? Because office documents do not pass a handle to these damn record & play tools! What we did was to thereby use office automation and effectively automate out all those 300 tests!!

I don't really know what the moral of this story, but all I care is that I got rid of those 300 tests for the rest of my time in the project!! Btw, do you want me to tell you about the 2000 odd regression tests we had for the thick client version of the same query tool ?? :-)

It was great reading your blog anyway! Here is another thought for you to chew on, softwares automate an otherwise manual process but that doesn't mean that human knowledge and intervention is unimportant, but that also doesn't mean that softwares are not required! A clever & innovative use of softwares is what matters so that we can go home and have a good life!

Cheers,
Ajay

Pradeep Soundararajan said...

I don't really know what the moral of this story, but all I care is that I got rid of those 300 tests for the rest of my time in the project!!

The moral is this: You helped the computer do what its good at doing and that gave you time and breather to do good at what you do [ thinking and exploring ]

It was great reading your blog anyway! Here is another thought for you to chew on, softwares automate an otherwise manual process but that doesn't mean that human knowledge and intervention is unimportant, but that also doesn't mean that softwares are not required! A clever & innovative use of softwares is what matters so that we can go home and have a good life!

My reply above in this comment is finally what you are trying to say and that's OK.

BUT both the automation and humans are fallible. If once when your client needs information and Office Automation fails that creates a loss to a stakeholder or customer might make those 300 come back to you because people can ask you, "Why did you do it badly" and they can't ask a software that question.

You must be careful in understanding in what context you got rid of the 300 tests and in what context you did not.

That's why the term "Computer Assisted Testing" rocks.

Dumitru said...

Nice post, and very well said: “Automation replaces humans - The truth about what kind of humans it replaces”.
Cheers.

amar said...

Pradeep you really made a nice post that most of the people who thinks that test automation replaces the manual testing.

As a matter, Automation adds the skills to tester, but it will never replace the manual testing, cuz sometimes we need manual testing where we cannot perform some tasks like writing the scripts, or test cases ... etc. and lot more stuff. A small example Every one uses Word, but not all uses spell checker, which can be used for checking the wrong words, Using spell checker is also something like automating, Though the test automation is used, the scripts have to be written manually, or though we record then also recording is done manually. Manual testing can never be replaced by test automation, infact test automation is like a coin contains both test automation on one side and manual on the other.

Test automation is not the future. Where i also want to mention one of the point "vendor locking" which is the greatest disadvantage of test automation.

Thanks,
TesterWorld

Pradeep Soundararajan said...

@Dumitru,

Thanks for the comment and for liking the title.

Anonymous said...

what bung!you must be one of them wanna-be-tester phony who talks shit to save your sorry ass from getting sacked (obviously, for the lack of any real knowledge about testing!!) its a shame that I came across your post!!

Pradeep Soundararajan said...

@Anonymous,

what bung!you must be one of them wanna-be-tester phony who talks shit to save your sorry ass from getting sacked (obviously, for the lack of any real knowledge about testing!!) its a shame that I came across your post!!

This blog of mine is to help people like you feel ashamed and guilty. I am glad to know it is yielding it.

If you think you are saying the truth, you wouldn't have chosen to be anonymous but I am sure you knew you were a coward.