"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Tuesday, April 17, 2007

The negatives of "negative testing"

Before you proceed reading this post, please take a piece of paper or a notepad and write down what you mean by "negative testing".

Are you done with it?

I spent time investigating what many Indian testers say about "negative testing" by browsing online forums, google search and I also sought help from testers who were online in my buddy list. What you read below till you see a "Meep" are views of testers who responded to me over IM when I asked them - What do you mean by "negative testing"? or When someone says "negative testing", what occurs to you? or What do you think about "negative testing"?

"I infer that some of the people are trying hit on negative tests (the probality of user acting the same is very less) o begin with"

"Testing carried out to test expected behaviour from product by providing wrong (negative) inputs "

"A negative test would be the program not delivering an error when it should or delivering an error when it should not. Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to)"

"there is nothing called as negative testing.. but there is something called negative test case or test idea.."

"Negative testing is that testing which attempts to show that the application does not do anything that it is not supposed to do"

" input which do not lie in valid domain"

"
any testing carried out by passing non reccomded values with an aim of breaking down the application is called negative testing"

"if a developer designed a edit box to accept only numerics and length of 10 nos and if it accepts alphabets which we enter if it accepts either than numerics is negative testing"

"negative testing is checking for scenarios which are not smooth path.."

"
Testing aimed at showing software does not work. Also known as "test to fail".in simpl ewords"

"negative testing is something you know tht the values wont support but u need to test it.."

"negative testing for me will be done after validating the postive test cases first"

Look this is an interesting one - "as far i am concerned..i belive there is noting called -ve testing."

Meep!

I am happy that Debasis and Venkat Reddy asked about contexts before they came out with their answers.

I found a lot more definitions by individuals in online discussion forums and orkut communities about what people had to say about negative testing and it's a similar experience to what you have read above. If you are interested Google it up and have fun!

Thinking critical, I conjecture that testers to my knowledge are using the term "negative testing" as a guide word heuristic. They are happy of using it because they found bugs with it, be it whatever they did assuming that they were doing "negative testing".

So doing something that could find bugs is more important than what meaning one has for the technique he is using to test ( in the context of this discussion) BUT testing in a skilled way is more important than just doing it. The problem comes when you have to describe what you did to others, especially when they question the techniques you used. A "stress" test to you might be a "load" test to someone - How would you communicate yourself effectively being in a land of confused terms?

After I wrote the above sentence, I leaned back on my chair for a while and thought the following -

"Negative testing" might have different meanings in different different contexts and the following list are my assumptions of the scenarios where I might have done or might do negative testing -

  1. If I find a bug but did not report it, which later turned out to be a critical issue, I did negative testing.
  2. If I fail to observe a bug that passed away the screen, which could have been caught if I had practiced my observatory skills, I did negative testing.
  3. If I do not learn the product or technology during test execution, I did negative testing.
  4. If I do not run cost v/s value in mind and end up doing an expensive test away from the mission, I did negative testing.
  5. If I failed to ask questions before accepting the mission or jumping to test, I did negative testing.
  6. If I blindly follow the specification document, believing it's a bible, I did negative testing.
  7. If I am into automation testing just because my friends said "Manual testing has no future" without realizing that automation testing also involves manual activity, I did negative testing.
  8. If I manipulate the metrics, without realizing the impact of it on reputation of the company and myself, I did negative testing.
  9. If I freeze my test plan and never bother to change it despite the daily dynamics, I did negative testing.
  10. If I mess up my relationships with developers, without realizing the fact that developers are great resource of information for testing, I did negative testing.
  11. If I stopped learning or practicing testing skills, I did negative testing.
  12. If I spend most of the time in orkut and online groups asking questions about how I could solve my day to day problems with subject "Urgent - Need help", I did negative testing.
  13. If I list everything that I have in mind about this topic giving less room for you as a tester to think about this, I did negative testing.

Some testers expressed shock when I said to them that I don't know what negative testing actually means. It's very important to say "I don't know", in my opinion when we really don't know something being asked. How long has it been since you said, "I don't know"?

I recently faced a question from someone who claimed to be my friend but isn't happy of my growth in testing field, "Are you a humble person, Pradeep?".. I said, "I don't know" and he replied, "Well, this answer itself says, you are not". I had to laugh at three things - the situation, myself and then my dear friend.


Not knowing a good enough definition that suits your context while applying it, is a negative thing to happen to the project you are working or for people who are tasking you. When you do negative testing with already a negative happening, you might be finding bugs because the resultant is always positive, mathematically. Ah! That's my imagination and I could be wrong.

I am open to more ideas on this but if you put the comments in a negative way, I might have to make the resultant positive ;-)

Testing in my opinion, should not be classified as positive and negative, it isn't electricity but still someone has done the damage by misunderstanding someone else. I don't know what I did with this post!

-- Pradeep Soundararajan - pradeep.srajan@gmail.com - +91-98451-76817

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

22 comments:

Ben Simo said...

Hi Pradeep,

Thanks for the great example how words are often just heuristic models of ideas and that they can be interpreted differently by different hearers and in different contexts.

When I think of negative testing, I think of testing that seeks to determine if the system under test fails to meet a requirement by testing the negatives of a requirement. I recently posted an example of this in my blog titled Get excited about the negative. Perhaps it was due to my recent thinking about, discussing, and researching the impact of selecting thinking on testing that I immediately went to that definition of negative thinking. And that too demonstrates how confirmation bias can impact our communication with others.

I must say that I like your examples showing "negative testing" to be testing that has a negative impact on a project. I never knew that "negative testing" could be as easy as failing to report a bug. :)

Cheers,

Ben

Lalitha said...

Great thoughts Pradeep. You make your readers start thinking.

SOUMIT said...

There is nothing called -ve testing except for purpose-less testing (or like pradeep said, 'failure to report bugs etc). What you have is -ve situations for the system that you want to test; hence, -ve test scenarios/cases etc.
So i've got to test that system does not allow to delete saved records. (No, two -ves don;t always a +ve make!). The situation is a -ve one but the outcome is +ve(or -ve if the test fails).
what say?

Pradeep Soundararajan said...

@Soumit,

There is nothing called "nothing" but still we use it and similarly, I feel there is something that people have been calling "negative testing" maybe without understanding what others mean by it.

I agree that situations too are negative, most of the times but the outcome is positive sometimes but not always.

Opportunity is missed by most people because it is dressed in overalls and looks like work.- Thomas Alva Edison

And so, it depends on the person who is dealing with a problem. If he is unable to handle it, he claims it to be negative and if he handles it and boosts his confidence, he thanks the problem for giving him an opportunity.

@Lalita,

Thanks! I should keep doing it and that's the challenge ( or opportunity )

Joy Forever said...

It was a great write up Pradeep. It was indeed a new way of interpreting the term "Negative Testing" as testing that has negative impact on the project.
We often used the term "Negative Test Case" in our project. It usually meant a test case that tests whether the application behaves in an expected way when unexpected input is given. I would like to point out that many developers and even managers are extremely reluctant to encourage such test cases, and in some cases we were told to drop such tests because the scenarios they tested were "too improbable".
Thanks for sharing your ideas. Keep up the good work!

Anonymous said...

Testing is to verify the User Need. But -ve testing or -ve inputs are used juz to make sure that the System can withstand any kind of Input. It is normally considered as Users of any system may enter any kind of values but the system to respond with a error message. If a system gets an input feed then it should be able to handle the corrupted data also. So negative testing becomes important. And some times the +ve fails after a -ve testing juz becos the exceptions are not handled properly. So -ve testing have its own importance. What do you say?

Pradeep Soundararajan said...

@Anonymous,

Testing is to verify the User Need

Which end user? Ram or Shyam or Gita or Sita...?

Testing is a questioning a product in order to evaluate it - James Bach

I subscribe to the above definition of testing from James Bach because I realize that every test I do is a question I ask to the product.

I do not want to comment on your view of negative testing because as you see above every person has their own definition or view about negative testing.

Anonymous said...

Hi Pradeep,

It's a thought provoking write up & congrats for the good work.

But then
1. What's the mission of this post ?
2. The list of scenarios shared here help testers to improve their productivity. Is that what you are trying to convey ?

Pradeep Soundararajan said...

@Venkat Reddy,

Thanks!

1. What's the mission of this post ?

As I am the owner of this blog, I set a mission to myself - to write a thought provoking post.

2. The list of scenarios shared here help testers to improve their productivity. Is that what you are trying to convey ?

It is your view of looking at it and many testers look into each post of mine into different views and learn or like different things from the same post.

I realize that I am also one among blog readers of Tester Tested and I enjoy reading it.

Anonymous said...

I just have one word to say about your blog "Awesome"

Anonymous said...

@ Pradeep,
I agree with the words of James Bach but the who is a Tester.

Tester is one who thinks in the perception of the User also assures the quality of the product.

It is very much necessary to evaluate the product. But it is very much necessary to think in the angle of End User(Ram or Shyam or Gita or Sita or Pradeep). Your comments pls. Correct me If I am wrong. Thanks!

Pradeep Soundararajan said...

@Prabhakaran,

it is very much necessary to think in the angle of End User(Ram or Shyam or Gita or Sita or Pradeep). Your comments pls. Correct me If I am wrong

"Quality is value to some person" - Jerry Weinberg

What is bug to me, isn't a bug to you and hence it's impossible to satisfy everyone or think like everyone. We can list names - Shyam/Ram/Gita/Sita or even Pradeep/Prabhakaran but neither I nor you nor any other tester is able to think how any of these people think. I myself have to admit that I don't know how I'd be using any product that I am currently using, next year or even tomorrow.

Apple sold their 100 millionth iPod last week. A friend of mine doesn't like iPod. Apple might not want to satisfy him because he is just one person but if the whole India dislikes iPod design then maybe Apple might think of re-designing it to see a good sales in India.

I was curious on the topic that how do people design and test their products that are liked by millions of people and I found a company called Frog Design, who designed the Apple iPod and a lot of other gadgets and web pages of Microsoft.

The team in Frog Design - consists of artists and psychology experts.

Which tester to your knowledge has learned or practiced psychology or study of how people think?

Many many testers to my knowledge see software testing as a non critical thinking activity.

Scripted procedures rule a product test and critical thought process takes a back seat most of the times.

If you want to be an expert tester and is more curious to look at what an expert tester like James Bach studies - look at www.satisfice.com/rst.pdf where one of the slide points to the subjects that James Bach is practicing.

Anonymous said...

Of course, I tester can't think how an end user thinks. But it becomes must for him to test the all possible scenarios or inputs that a user may enter.

A big concern may introduce people who know psychology.

The responsibility of a Tester is to understand how the business work and how the whole system will behave and what the real time user expects. For that he don't want to read psychology.

Ur comments plz....

Pradeep Soundararajan said...

@Prabhakaran,

The responsibility of a Tester is to understand how the business work and how the whole system will behave and what the real time user expects. For that he don't want to read psychology.

I think you might have missed seeing the slide I suggested you to read carefully.

How do you know "he don't want to read psychology" - let's talk about us - If you realize that by reading and practicing subjects apart from the theory of software testing makes you a better tester, would you read and practice it?

I suggest you to give a critical though about this.

Dinesh Karmankar said...

Hi Pradeep,

Testing in my opinion, should not be classified as positive and negative, it isn't electricity but still someone has done the damage by misunderstanding someone else.

As you always say, any testing activity should be context based and Negative Testing is no longer exception to this. We know that testing activity(s) brings some output/value and to measure this output, we use adjectives like good/bad/excellent. I think same applies to Negative and Positive testing too with the difference of perspection looking towards it.

I don't know what I did with this post!


Thanks for such a nice Negative Thoughts for Effective Positive Testing.

Ben Simo said...

@ Prabhakaran


Tester is one who thinks in the perception of the User also assures the quality of the product.


Testers need to test from a number of perceptions. If a software product makes users happy but doesn't bring in revenue, that may not be a high quality product -- unless making a profit is not part of the context. Testers need to put on a number of different hats when testing.

Also, testing cannot assure quality. Testing can help measure quality, but testing does not fix quality problems. Testing only provides information. Its what we do with that information that impacts the quality.

You can't test a bug out of a system. :)

Ben
http://qualityfrog.blogspot.com

Suresh said...

Pradeep u rock in ur thoughts man.I am reading ur blogs.Its simply awesome, i come here just to make sure atleast i learn one new thing. :-) Keep posting...

Rahul Verma said...

Hi Pradeep,

This is a nice post.

As for me, I have never used the term "Negative Testing" as such, though I talk about negative Test cases, as the test cases which solved two purposes -
1. Inputs which make the application throw a customized error
2. Inputs for which the application did not plan anything - neither an error nor acceptance - it just behaved in an unexpected manner.

Your post in explaining "Negative Testing" in the context (as I thought of it!) of Inefficient and Ineffective Testing, is a good lesson. I agree with you on the fact that a tester should study the sujects which can make him a better tester, though the subjects may not be directly related to software testing. I found "Usability Engineering", "Information Visualization" etc. as subjects which might be of such help and I have started working in this direction. Infact, I had presented on User Behavior and Performance Perception Analysis, based on my study and analysis.

Thanks for sharing your wonderful thoughts.

Regards,
Rahul Verma.
http://testingperspective.blogspot.com/

Anonymous said...

Hi Pradeep,

"there is nothing called as negative testing.. but there is something called negative test case or test idea.."

I also agree with the same. By creating negative test cases I try to break up the applications.

The one thing I would love to mention is that your choice of contexts, examples and topics for discussions are Awesome.

Thanks for giving us very good thougts for thinking.

Thanks
Mukesh Dhingra

Unknown said...

Hi Pradeep,

This is the first time I been to your blog and this is the first post I read.
This is really a different way of thinking.
It is really of great help
Thanks
Harshita

Anonymous said...

There is nothing right or wrong in using the term "negative testing". There is another term called "domain testing", we can ask the same question again you would get several answers.
I think "negative testing" simply means in any in test/tests if we were to provide inputs that are perceived to be negative then that test is called a "negative test".
If there are boxes in which we are to give numbers to add, if instead of providing numbers we provide special characters then, that would be a negative test.

- Sam

sunil said...

Great Article Pradeep Thanks