Educating customers on testing

Everyone has heard about testing and unfortunately everyone has heard about it from different sources that say different and contradicting things.

• One of the customer I worked for thought - testing is about delivering bug free product - which, in my opinion and many other experts with whom I interact is an impractical idea about testing.
• Another customer I worked for thought - testing is about delivering quality - which, again, is a bad idea.
• Another customer thought - testing is about executing test cases and based on the test case pass/fail ratio decide to release - even if the test cases isn't helping testers catch bugs.
• Another customer thought - testing is a job of repeating tests to compare build quality and take a decision to release a specific build to his/her customer.
• Another customer thought - testing is meeting specification.
• Another customer thought - testing is satisfying the customer.

Working with multiple customers is a huge challenge for test managers or testers like me and you. I have so far been repeating the word "customer" and whom do you think I am referring to?

I am referring to the manager we report to and the manager we report to is also our first customer. When testers quit their job and join another one, they never ask their customer (manager) what he/she thinks about testing. That's why some people feel they were performing good in their previous company or they are able to perform better in their recent organization - reason being - their idea of testing matched with that of their manager's idea.

If our idea of testing doesn't match with that of your customer's ( manager ), you and I seem to face a lot of trouble in meeting our customer's expectation and are unable to satisfy them.

Everyone, including you and me might be thinking that the idea we have towards testing is the right one. I have been realizing and teaching that there isn't THE RIGHT WAY to do something. Rapid Software Testing taught me cost v/s value and I took it very serious and I am practicing it.

All customers, irrespective of what definition they have towards testing look for cost v/s value. I am striving to deliver a fabulous value to my customers and I am facing a challenge since I realize what value means to me is different from what value means to my customer (manager).

That's why I was looking as a bad tester (despite finding a lot of bugs) to some of my managers in the past. They think the value they wanted to get out of me is - to execute thousands of scripts or test cases.

Unfortunately I wanted to add more value to the testing efforts which wasn't liked and was fired for attempting to do that. They weren't wrong - they just cleared a trap that they dug in when they hired me.

I learned - what I think as value might be a threat to someone else's idea of value and I value this learning I had about value.

Once I (and my team) found a lot of bugs, so much that there weren't as many bugs that the test case document had helped the team catch in the past. The questions from our customer started to focus on why we had a bad test case document and *not* "Why can't we do more of what you guys did?"

A definition, customer has towards "value" doesn't appear to change despite showing enough evidence. Its hard to continue working for such a customer but there is another dimension to think - money.

Some of those who attend my Exercises for a testers mind - A Rapid Software Testing Approach workshop say, "Well, this is great. I am excited to do testing this way but I suspect the problem is - I don't think my manager would be interested on any of these things I do"

I have had different replies to the above:

• Stop thinking that your manager would not be interested. That's the first step.
• Convince your manager to take this workshop.
• Educate your manager on this.
• Just do it and make your manager ask, "How are you doing these things?"

Here is a list that I suggest you to use, if you realize facing similar challenges: (I practice the same, too)

• Educating your customer, starts with you.
  
• Stop thinking that your first customer wouldn't be willing to see more value.
• Provide evidence for him/her and ask her to critique the evidence you provide.
• Request for a discussion about - What you think of value v/s What your first customer thinks of value.
  
• If it fails the first time, provide repeated evidence and suddenly do things the way your first customer wants to do - see if your first customer comes back asking why suddenly things appear not so good.
• If you have been successful educating your first customer, then motivate your first customer to educate his/her first customer.

BTW, what do you plan to educate your first customer on?

Here is a list of heuristics about educating your first customers:

• Educate yourself before you educate your first customer.
  
• Testing is questioning a product in order to evaluate it AND testers provide quality related information to stakeholders/management to help them take better informed decisions -- James Bach & Cem Kaner
  
• 100 test cases might not help a tester catch 1000 bugs and if you want more bugs, exploratory testing ( with a mission/charter) is an important approach to be incorporated. Adding another 1000 test cases is not a solution.
• Specification document is just one or the oracle that helps a tester find bugs. End users or your customers never use specification oracle to spot bugs and hence going beyond specification can help in finding more bugs.
• Session Based Test Management is a way where Exploratory Testing can be tracked and managed more efficiently.
• Spending less time on documentation that's going to end up wasteful is a nice way to get more time executing tests to find more bugs.
• "If you are too bothered about repeatability of tests - then it is good to have a document of test ideas and *not* test cases that tends to have more ambiguity."
• "Diversification of approaches and techniques helps testers find more and different bugs that otherwise might be caught by a customer."
• Software testing books are not the *only* books that helps tester find bugs and hence the library should stock a lot of books on thinking, ideas, pattern, philosophy, psychology, humans, communication...
• We didn't appear to find a bug because of one or all of these:
  
• We did not have the skill.
• Someone did not let us know when they put it.
• A stakeholder asked us not to report it although we found it.
• We didn't intend to find that.
• We thought it is OK to not to find that, if it existed.
• We were finding other important bugs thinking opportunity cost.
• We didn't have the right oracle.
  
• { You might want to try a lot more, provided some of the above works for you. Sometimes I am aware that I am going to fall into a trap but that's a way I choose to avoid bigger traps }
  

There might be a trap you may fall into. Some of your first customer's might not deserve/ might be reluctant / be egoistic - to such an education, sense it out smartly and stop educating him/her to get more time to educate yourself. An indication that you fell into a trap is when your first customer fires you.

+ A heuristic is a fallible method of solving a problem
++ An oracle is a principle or mechanism by which we (humans) identify problems.

--
Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it." --

Jony Jony, Yes Papa! Following Process, Yes Papa! Telling Lies, Ha ha ha!

I am 100% sure that this article would make no difference to the world because there have been better insightful articles than this about the topic that I have started to write, which didn't make much difference to the part of world I think I am living in.

I feel it is important for you to ask yourself, "Why would I want to read an article that the author is certain that it wouldn't make things around me better?' because by asking that question and continuing reading ensures that you are wasting your time because you chose to.

I conducted my public workshop on December 1st through Edista Testing - a QAI venture in Bangalore on December 1st. A couple of months ago, I had expressed my dream to go around places in India other than Bangalore and conduct my workshop on human skills of testing titled Exercises for a Testers mind - A Rapid Software Testing Approach and the dream is coming true as Edista plans to bring this workshop to your city wherever you are in India. Do not worry about the cost as long as your company can sponsor you for a day.

I challenge testing minds and I learn from them and some of the curious minds who attend the workshop largely benefit from the workshop. All audience are thrilled about the things they learn from the workshop but what makes them sad is that they have to go back and follow the process their company mandates them to follow - which at the end of the workshop they know that it doesn't add as much value as they witnessed the value that Rapid Testing adds.

How do you think they witnessed, the value that Rapid Testing adds V/S the value that their process ( IXX, CXX - Level X, SICK6 SXXXX, XXXXX, Centre of Excellence, Test Factory) adds?

This time I felt terribly challenged at the question of process and I revealed something that shocked the audience and were not willing to pose further challenge. Its exactly an year old secret that was lying in my inbox and I had shared it with very few.

Rapid Testers stand up to scrutiny and so I stand up to scrutiny in case the following information you might read, appears to be an exaggarated or made up to prove a point. It might appear so because those who dwell with those processes can never achieve it and it is close to impossible for one who believes in those to believe the following.

Here is an e-mail that I received last December from my Supervisor from a large Products + Services company of India on the last day of my job at that company.

---- Forwarded by Pradeep Soundararajan on 12/27/2006 04:29 PM -----

All the best, Pradeep.

There are learnings for all of us from the way Pradeep has conducted himself in this tenure in the team. The confidence he has shown and willingness to help others and constantly exploring for new ideas are some of the highlights. Fortunately I was also part of the team which Pradeep was associated.

One more thing I would wish to share with you all is that, he was handling XXXX (product name masked) releases for XXXX (A multi billion dollar customer name masked - who also had a test team at his end who tested the releases we made before they released to their customers ) and with proud I can say that there isn't any bug which customer found apart from what we or Pradeep found here.

On behalf of the team and on a personal note I wish him all the best in all his future endeavour .

Regards
XXXXXXXXXXXX ( Supervisor's name masked )

__ end of e-mail excerpt __

Some very important points to think and remember:

• To remind you, this company too, is one such who believed in the above mentioned process till I helped at least some teams realize that they could achieve bigger success if they could come out of the trap they fell into and add more value to customers. No customer would say - I don't want you to add more value but I am paying you to follow XXXX process. If a customer says that, either the customer needs to be educated on testing or it is a customer who deserve to be off the business list unless the customer is willing to pay a huge price.
• Some of you might think this post as my self marketing and might fail to learn some important lessons that your customer might want you to learn. Also if I wanted to use the above as my marketing, I wouldn't have waited to reveal it an year later.
  
• Some of you might think this happened somewhere too far away from India, which is not true, because this happened in Bangalore, India.
• Some of you might think I did *complete* testing but I admit that I know no one can completely test anything, so I did not do complete testing.
• It was no one man Pradeep show, it was a team effort and it was achieved because I practiced the skills that James Bach and Michael Bolton helped me gain on the product I tested with the skills that I already claimed to have + the skills of the team + Exploratory Testing mixed with very little scripted testing.
• This achievement for the company, the team and me didn't happen because we intended that to happen at the start of the project BUT we had realistic goal as a test team "To find important problems, quickly, and gather as much information as possible through our SKILLS and present them in a useful manner to help the management take informed better decision" AND NO STUPID GOAL SUCH AS "LETS MAKE A BUG FREE PRODUCT".
• By the customer not finding any bugs other than what we found, although there was a test team at his end, doesn't say that the product was bug free. What customers looked at is value for the money he paid us, and I think he was sure, as the project progressed that he was getting a bonus. At least it helped him understand that he had to hire a better test team than us to find bugs out of what we had found.
• It is hard for me to know what is the state of the product now but I can safely say that the 8 month duration I stayed there, there was nothing that our customer or ever his customer reported on the releases we made that we didn't know.
• I'd like to add another important point that in order to achieve what you read above, I and the team had to break certain things that the process mandated to achieve it.
  

Don't ask your management or customers to read this post because both of them might start to demand more from you, which might take away your coziness you have been enjoying in testing with the process you are asked to follow. In case you are curious to share with them, be prepared to learn and practice human skills of testing and better insight into testing, out of which some of them are taught by James Bach and Michael Bolton's Rapid Software Testing and Cem Kaner and James Bach's BBST courses.

"I would say that a process is the way things happen. The Earth in orbit around the Sun is a process. In that sense, yes Rapid Testing is a process.

What Rapid Testing isn't is a set of instructions to be followed without understanding. It is not a collection of physical behaviors. It isn't even really a set of techniques, although it does feature some of those. To say
it's a skill set and mindset is to locate RST within the mind of a tester. It's a process of making sense of testing problems and reacting to them differently, as the situation demands." -- James Bach


"One of the hallmarks of Rapid Testing is that your work can suck less. The pointy-haired bosses can tell you what to do to some degree, but they can't supervise you every minute of every day, and they can't tell you how to think. In the "free" time that you have--those little micromoments of disposable time, for which you won't be punished because they can't watch you every moment--you can perform a quicktest, find a bug, write another line of the test tool you're working on, sneak a look at the specification you managed to photocopy, check in on your mission, befriend a developer, have a chat with another tester about where the bugs might be, cover a product just a little more deeply with just one more test... And after you've leaked a little of what you've discovered and have a few successes under your belt, maybe you can start to train your manager into expecting nuggets like that." -- Michael Bolton

Here is another evidence of high value addition that Rapid Software Testing produced to Michael Bolton's client

Here is one of the great and very insightful quote I have read about achieving the mission, "Who cares if you followed the instructions if it doesn't work when you're done?" - Scott Barber's Dad

All I have done in this post is to complicate that quote in many words which the wise man said it in one sentence. I apologize for that.

-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

How to show product quality? - (Uncensored Version)

The most challenging software testing quiz

This quiz created by me challenges a tester to answer several common and uncommon questions. At the end those who finish the quiz get to know what prize they win. ( which means, there are a lot of prizes to be won)

Last time I announced I would give free books - Lessons Learned in Software Testing and I did give it to those testers who proved to me that they deserved it by working on an exercise I gave them. This time I am keeping it as a secret because those who finish the quiz will get to know their prize when they see their performance and results. I have some questions that are easy, some which are moderate and some that only experts can crack - so whoever you are - you have a challenge.

Update me your score because if your score is more than other testers who have taken this quiz then you have a chance to win the many hundred dollars that I plan to give as a bumper prize. So here is the link to The Most Challenging Software Testing Quiz ( I could ever think of creating)

Good luck!

Update: 12th October 1300 IST: The quiz is updated with HTML formatting and looks more great to my eyes. Thanks to Adam Goucher for his suggestions. 3 more very challenging questions are added. So you might want to re-take and check your score. If there is any betterment of your score update me so that you dont miss the grand prize.

Update: 12th October 2345 IST: If you dont see your comment for this post it is because I felt your comment contained some clue about the answers which might distract new people taking the quiz. In case you want to blog about this quiz please make sure you don't give the answers or let the people know the answers or experience as they might lose the excitement and opportunity to learn.


-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton

Notes from spying test experts

This is a top secret post and be very careful with the information. I am not sure if I can expose my notes to my dear blog readers but I am taking a risk of doing it.

There are some secrets of test experts that not many know. I had a mission sometime back to spy about how test experts actually find many important problems, quickly.

Many people think ( that included me, too) that Test Experts do lot more tests and have more fantastic test ideas and that's why they find many important problems, quickly.

I was hoping that there is more information hiding than what is visible about test experts and spying helped to uncover those hidden secrets.

This post is all about the secrets that I found spying test experts and the secrets of how they find many bugs?

You can trust me about the information to an extent that I and many other testers have been practicing the secrets that has resulted to our success of finding many important problems, quickly.


They just do one test to find many bugs!

Now, I am sure you don't want to believe me because my finding says that they do just one test to find many bugs. I am sure you have seen testers finding one bug per tests that find a bug and that makes you not believe this information but... you must understand that the secrets are always unbelievable.

You can feel safe after knowing a fact that I too didn't trust the secret of one test finding many bugs but my spying mission was to find out more granular details about it.

Looking more carefully into it, you might discover that it is not a test that finds a bug but it is a human that finds a bug and a test plays a role in helping the human find it. So there is something with humans that is helping to find bugs and not completely with the tests.

When I collected this much of information over months of spying, I was invited to be a part of the round table conference of experts as I had posed myself as a test expert from India (I knew I wasn't a test expert and even today I am too conscious that I am not one such) but you know... I am a spy and had to pose that way.

Over the round table conference, test experts started discussion about ORACLES* and I didn't know their terminology and just took notes. That evening, when I returned to my hotel, I was trying to connect ORACLES and the spying mission I had in hand.

* Oracles - principle or mechanism by which we (humans) identify problems

I was perhaps lucky that James Bach and Michael Bolton were staying in the same hotel that I was put up and we caught up at the dinner table.

I had an idea that I hope it worked. I was hoping that James Bach is a kind of person who would spill off the secrets in a context to prove me wrong and help me learn the secrets if I talked nonsense about ORACLES, because his passion to the craft of software testing is something that I am trying to match. Michael Bolton, the Birbal of North American software testing was a person whom I had to handle very carefully as he is more likely to identify me as a spy. At the end, I assumed that I was able to outsmart Michael in not letting him know that I am spy.

Going back to the hotel room, I started writing my discoveries about the secrets of test experts and the granular details of how they find important problems,quickly.

Before I publish a report to end the spying mission, I was sure that I had to practice the details I collected. I did practice and was excited that those ideas really worked and I found more bugs by executing one test.

Why many testers might not be able to practice the secrets that I am unlocking here is because they have a fixed expected result that is derived from a specification document that the tester thinks is The Holy Bible for the project.

Specification Document - is one single oracle and hence you might find one bug per test you execute.

The important thing is experts define a bug as "anything that threatens the value of the product" (-- James Bach)

Here are a list of Oracles that experts use with every test they do:

Consistency with the Image: When I execute a test, I try comparing and contrasting it with the image that my company or stake holders have been projecting about the product or the company.

For instance, if my company has an image in the market as people who produce software whose performance is good, and I execute a test whose documented expected result is "This functionality should work", I ask question, "Well it works but didn't it take too long to perform the operation?" and then I find a bug and say, "Although this operation goes through the time it took to complete the operation exceeds beyond the image the market has about us"

Consistency with Similar Product(s): When I execute a test the functionality might work as expected but I ask a question, "Well it works but doesn't it seem to perform in a way that users who are used to similar products might expect it to happen?" and then I find an important problem, quickly.

Consistency within the Product: When I execute a test that appears to have passed, I ask a question, "Well it works but doesn't it work in a different way than other areas of the same product?" and then I find an important problem, quickly.

Consistency with Claims: Marketing personnel and my manager are making certain claims about the product. When I execute a test and the test appears to have passed, I ask a question, "Well it works but does the way it works go against the claim that they make?" and then I find a bug.

Consistency with Statutes: When I execute a test that appears to have passed, I ask a question, "Well it works but does the way it works go against a law that an organization or a certifying authority might have?" and then I find a bug.

Consistency with User's Expectation: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in a way that the users whom I am aware of might not want it to happen?" and then I find a bug.

Consistency with History: When I execute a test that appears to have passed, I ask a question, "Well it works but does it work in the way the previous releases of this product work?" and then I find a bug.

While writing this post, I got an e-mail from James and Michael that reads...

Dear Pradeep,

We knew that you were spying us for a long time. We also saw the way in which you took notes, asked questions and drew inferences from what you could capture and that impressed us to help you learn more about testing because those are some important skills that a tester needs to have, to add good value for the cost customers pay for. We consider to hire you and we recommend you to quit spying profession and take up testing where you might get more laurels for your skills. Here is a link to know more about more secrets : www.satisfice.com/rst.pdf

Good luck!

-- James Bach & Michael Bolton

I was excited and I did quit spying, to take up testing as a full time activity. I thought I shouldn't post this in my blog because I didn't want those secrets to be given away for free and saved a draft of this post. I am afraid if Blogger has a bug that publishes drafts that are not intended to be published.


-- Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com

"Pradeep's first language is not English--his first language appears to be testing." -- Michael Bolton


Update : Michael Bolton, blessed me with a post on his blog for this article of mine. I was jumping in joy for at least half an hour reading that and I am pleased to share it with you: http://www.developsense.com/2007/09/if-test-passes-in-forest-and-no-one.html