Not that you don't know.
Truth About Test Plan Documents
- 98% of test plan documents that are created are not updated, maintained or cared beyond sign off.
- The first 5 pages of a test plan document contains history that doesn't interest even those whose names are mentioned in it.
- Scope of testing section is the most funniest part of the whole document. Some times when testers report serious problems, some people cite it as out of scope. Hey, its still in the product.
- Customers worldwide could have saved millions of dollars if their vendors didn't care about creating test plan documents.
- 4 years into a project, nobody knows about the test plan document.
- No matter how stupid or intelligent the test plan document is, testers still write the test case they want to write.
- As an offshore services company, writing test plan documents are a cool way of billing customers without actually doing testing.
- Every stakeholder feels a false sense of achievement the moment they have a test plan document irrespective of whether they actually have a test plan.
- The cost of reviewing a document that nobody is going to use it is really high.
- Those who think they are not ready to test because the test plan document is not ready aren't testers by any means.
- A simple maintainable test plan document is far superior than a detailed test plan document.
- A mind map is worth a thousand good test plan documents.
- Test Plan Document is a Document, not necessarily a test plan.
- The next time you ask a tester to write a test plan where she knows it is not going to be used or maintained, she is not going to put her heart in it.
- Some test plan documents are written in a way that it is obsolete in its first draft itself.
- Some reviewers of test plan documents aim for perfection. More funny stuff, they may not even know what the product is supposed to do.
- Those who know about opportunity cost are likely to write a better test plan document.
Not that you don't know.
Truth About Test Case Documents
- ~ 90% of testers haven't bothered to think why there is a "case" in "test case".
- For most people on earth, a test case means a test idea that is documented.
- The expected results column in test case documents are a copy paste of requirements document / stories. So much money goes into re-writing the requirements document into expected results column.
- If you are already laughing at test case documentation, you may roar to a bigger laughter at trace ability matrix.
- Most traditional testing services projects have 50% of their project duration spent on writing test cases. The team members in such projects complain unavailability of time to "actually" test the product. No wonder.
- Unless the context demands, detailing a test case is a sin.
- Detailed steps in test case documentation provided for humans to execute is something I personally consider as an act against humanity.
- More than 99% ( yeah, more than 99%) testers I have met have passed a test case (or a bunch of them) without actually executing it. It is so f****** boring.
- Test case documents bring more money to countries like India than what Bill Gates must have invested in setting up an office in that country.
- Those testers who don't know to test without test case documentation aren't testers.
- More than 98% of projects I have consulted in India didn't have testers doing "test design". Here is a way: Take the requirement and write at least two or three tests to "check" if that requirement can be marked a Pass. That is all the design that happens.
- Test case documents are actually "Check case" documents.
- If there wasn't check case documents, software testing as an industry would have attracted more talent and helped in building more passion for the craft.
- Businessmen love test case documentation. Testers hate it. Businessmen hire testers to write documentation. Testers trade their time for money, end up writing documentation for money.
- Test case pass percentage is a great way to fool stakeholders. People love to be fooled.
- I personally can write test case documentation for any buggy product and make it look like a bug free product.
- If all test case documents created so far were printed and burnt, we'd have fire for the next one thousand years.
- If you rate testers based on how many test cases they write per day, you'd always find people who can meet the number you want them to achieve.
- As someone said, "Testing at its best is, sampling". If you start writing and detailing the samples, you will have fewer samples than what you can have and you will never get to know about the product.
- If X test cases documentation takes Y hours, the amount of time spent on reviewing it and getting to sign off is 10Y. So, if X goes to 10X, we have 100Y hours of work spent on test case documentation.
- Some projects have great test case documents and no time to run them all.
- If you do a lot of documentation, you cant ship software, you can definitely ship documents.
- If you are hiring people who need detailed test scripts to test software, your hiring has ton of bugs in it.
- Those business people who ask testers to write "how long will this test case take to execute" and make estimations of test cycle complete time, should be executed.
- It is about Opportunity cost and Opportunity or Cost.
- No user has ever bought a product because the product was developed with lots of test case documentation.
- 99.999999999% of test case documentation I have seen so far doesn't care what the users really want.
- If testers read 1000000 words in a test case document the first time they were executing, they only read 10000 the next time and 1000 that next time and 100 for the next. Later, they don't need it.
- Some people think test plan document = test case document.
- The service most companies sell is test documentation, not testing.
- All good testers I have met so far, treat other testers as intelligent as they are and don't punish humans with detailed test scripts.
- Test cases don't assure repeatability of testing, at best it assures repeatability of testers getting bored.
- Funny that expected result of a test case should ideally be, "Software should go kaboom" BUT it is mostly, "We should see a boat sailing smooth as the day is bright and clear and the waters are not turbulent".
Just that, I know.
50 comments:
“If testers read 1000000 words in a test case document the first time they were executing, they only read 10000 the next time and 1000 that next time and 100 for the next. Later, they don't need it.”
One line summarizes all.
Also from my experience, I have always found that 99% of the bugs found are not from the test cases. The test cases that the testers write are too much obvious and the programmers must have taken care of those check cases during unit testing.
Writing test cases helps testers think about the product, a good way to learn about the product before the see the completed – but too much of importance and dependency on test cases are unhealthy for the product under test.
Ajoy Kumar Singha
http://testingcircus.com
Excellent !!!!
How I wish our client would read this ....
Writing a test case is a process. Ritual.
If process becomes bottleneck, change the process.
ISO says this..
Venkat
Plenty of truth in these statements, but I always fall back to test case writing for two reasons:
1. In three years, when the product is in maintenance and the feature needs regression testing. The future tester is not going to have access or time to deep dive. If I don't communicate my deep dive, I'm setting them up for failure. Test cases let me learn from my work.
What better way for me to tell him/her that there is a special case if the husband is from Brazil and the wife is from Canada, but they're living in the Dominican Republic? Test cases are a great way to capture domain knowledge.
2. Test case writing is like preliminary exploratory testing. For every 10 test cases I write, I have 5 questions. Those answers refine the requirement and I've just prevented a bug.
It's easy to overdo it, draw moronic conclusions and otherwise mistreat test cases, but I haven't found a better way them test cases to deal with the above.
Great post. One to remember and return to in future (possibly for some cartoon inspiration).
Here's my favourite part:
No matter how stupid or intelligent the test plan document is, testers still write the test case they want to write.
I agree with @qaDog 2nd point,
The problem (at least in test cases) is finding the right level which will serve as guidelines, and enable practical review which will enable good brainstorming.
over than that, it's a writing and maintaining nightmare.
Focus on writing Purpose,
Use as many tables as possible,
Keep equivalence classes as generic rules instead of specific values.
@halperinko - Kobi Halperin
Was asked to look at test plan of a testing project. As usual it was copy paste document which had the other test project details.
Sat together with a tester and make outline of plan with 3 pages. Then it was picked for further work. Yet the same copy paste document went out for whom it has to go.
I did know what to tell for this; not for sending the plan that came out of brain storm session. But for not being of help in making self to create the plan on own for the context.
Later shared my request for making use of copy paste test plan that was sent.
Great post, I've recently shared my own experiences on writing Test Plans, I believe that we are thinking along similar lines. It would be great to hear your opinions!
Post at: http://qaffeine.com/2011/11/17/lean-test-plans/
Thank God... You are here pradeep. I'm arguing with my wife for the last hour about the pitfalls of "over-emphasis" on test case writing and maintenance.
Thank God you are here as an example!
She keeps thinking about risk mitigation, what happens if a test leaves a company or refuses to give a proper KT, then test cases are the only hope.
Finally convinced her. Now need to show her exploratory testing (and for myself too).
There are so many articles, it's hard to find documents on exploratory testing for beginners or people who have been subverted by the indian testing culture
Any good links or articles or stories I can share with her
@Keshav,
Wow! You and your wife are in testing? Don't get into schools of testing clash, especially with your partner.
The first time I met my wife I talked so much about testing and she was disappointed. I promised her that I won't do that again and have been keeping up with it very well.
BTW, you may want to show videos instead of articles. Search for my Viddler Exploratory Testing Demonstration videos.
First of all, a big thumbs up for this post Pradeep. Always been a fan of your articles and the way you've evangelized testing.
Here are my two cents on this topic:
All these so called test plan/test case documents are nothing but means to confuse the testers and prevent them from doing actual testing.
In my experience, testers are always given a step motherly treatment even though the product never gets launched/released unless it's given a green signal by the test team (which unfortunately is forced most of the times - the usual 95% pass rate line of reasoning). The actual scenario looks like this - one guy writes the said plans/documents, another one executes them, and a third guy passes the cases, without even confirming whether they've actually passed or not and the end result is a bored/demotivated tester, badly screwed product, and a smiling test lead (you see the pass rate was >95%).
No wonder then that the guys who want a taste of real testing head to uTest and moolya ! :)
Good work!
Although all the points mentioned are right, the whole point of having process is to make working within a team of people with different personalities and backgrounds easier and improved. Its entirely wrong to say that everything that we maintain in documents is a waste..
@Nithya,
I think you are right. Documentation of process or documentation per se helps in dealing with different personalities and make operations smooth.
As an example, ours is a joint family and as you know in a joint family people do possess different personalities, so to sort out the problem, we have documentation. Now we are happy and things are running smooth. Without documentation our joint family could not have happened.
Sounds crazy? Exactly. That's how it sounds to me when you say documentation helps in dealing with people.
Hi Pradeep,
Yes, I'm right ! the whole point of documentation is to keep the show running (like the way joint families work).
And more important point is to raise your voice when some process is not making your work any easier and tweek it to help imporve the way you work. I guess that's not crazy? What is crazy is the team does it for compliance and surprisingly even leads/managers encourage it. In fact, when I started as a developer I used to be one of the people who hated documentation. But, all that a manager thinks of is "WHAT IF's"(I mean risks). He thinks of ways to keep the show running... That is his job !! isn't it ??
@Nithya,
So, what documentation should Rahul Dravid & Sachin Tendulkar do to keep India's show running?
Hi Pradeep,
When Rahul or Sachin are playing they are playing for India, for their passion and get paid for it whether they win or not. And that may not be their only source of income.
That's no comparison to an IT team working(to earn their bread) together to achieve something that their bread provider promised to some other company who spends money to get his job done.
If there was way we could do it all without documentation and management, professions like SQA(process only) and management would not exist.
All I want to say is do documentation for yourself and your team and more importantly read/update and share it with team. Only when the team gets the big picture, they'll put the best into their work. If it not helping you don't do it.
@Nithya,
If I ask you to develop a product for me and you estimate it will take 6 months to build it and you charge X dollars per hour and I see that half of the 6 months you spent in documentation, I could have saved that money to build two products.
Note: People are asking you to write test cases to make you a replace-able commodity. If you fall for it, you will never get paid a lot.
Sachin can't be replaced. So is Dravid. I want you to be rich. In case you don't want to be, I am sorry, I should not have cared for you.
Hi Pradeep,
A well described truth about test plan and test case.
It all depends on who/how the client see's it in spite of educating them.
Appreciate if you could let everyone how & what should be there in a test plan and test case (may be in a form of another blog).
This is for a wider audiences.
Rgds,
Gajaraj
Hi Pradeep,
You have very well explained the truth related to test plan and test case documents.
Clients/Customers have also realized this fact and have started implementing customized templates for test case and test plan documents. Also, they build a repository of these documents according to the type and size of projects. Whenever a similar project is initiated they suggest their vendors to limit their estimate of documentation based on prior experience.
However, this is only practiced by Clients who has a good QA setup at their end with well defined processes. This is not widely practiced as most of the clients outsource all of this to the vendor teams.
When outsourced, people who are decision makers and have eye on meeting quarterly targets tend to do what you have mentioned in this post. However, there are times when they meet passionate testers and then they have difficulty in getting wrong practices implemented. I wish we have more and more people like you in this industry who can challenge establishments and help deliver quality products to clients. Congratulations.
Thanks,
Vineet
Hi Pradeep,
Thanks for the post!
I might have enough time to take a closer assessment on the points above only after New Year. Just wanted to confirm any of them before it... so let's use this one: "A mind map is worth a thousand good test plan documents." Do you mean a mind map is not a good test plan document?
Thanks again and have a great end of the (Gregorian calendar) year!
Best regards,
Jari
@Jari,
A mindmap of test plan is worth more than a thousand good test plan documents written in traditional word doc. The power of visual representation is what I was trying to express.
great article
Even i faced hard time while executing test cases written by the testers who have worked on the project previously, and they are not just acceptance test case, i mean only describing the basic work flow, hard part was that,i need to write the "Actual Result" part by my on hand ( being a S\W professional its a shame) just because our client was conducting a external audit and it was an application related to health care domain , i executed more then 1500 of test cases and log the results manually ( in 2 months )By the time of doing this i use to think weather i am a engineer or a clerk.
@ashish,
Greetings!
Thanks for taking time to explain your situation. I understand what it means to get an external audit. It may appear to you that it is all about documentation and less of testing. Unfortunately, the lawyers don't understand or are unlikely to understand testing so they demand that much documentation being Health Care. Please google and find out how some company used Exploratory Testing and Session Based Test Management for Medical Device testing.
If you were to accept a little free advice from me, here is one: Your thought process appears to be like a tester but your writing appears to be like a clerk. Please make your writing skill consistent to your thought process of being a tester.
@Ashish,
Look at me: I learnt something after writing the previous comment.
Clerks don't mean they write poor English. My wife worked as a clerk and her English is good :)
As you compared a tester to a clerk, I just had to say it to help you understand the point about English.
We all go wrong, what matters is how we are able to recognize it soon and correct ourselves.
@ Pradeep ,
Thanks for accepting my comment and for the closer look :) , I am really feeling good that you pointed out some thing about my writing style, believe me it wasn't the first time , but now onwards I will take it seriously.
I used to think that as a Tester, finding out functional issues is only our prime responsibility( I hate GUI issues.. alignment,text size, colors.. etc. because I don't feel my self comfortable on having a deeper look on them ). Later on as my testing career exceeded I realized that we Testers are the ISI mark ( " Indian Standards Institution " mark of course ) on a software project , and we need to be good in documentation also. As we need to review and approve the FRS , SRS , IQ documents.
I did not take them seriously ( all I wanted was ... Give me the Application ...Show me the requirements , Tell me the environment )
Now I relies how Important these documents are , IQ is the soul of the application, and FRS is the Life story ( as in one sense), Test Reports for Audits. Test cases for the validity. So on..
I know there will be lots of other mistakes also presented in this comment, but that is why fellow Testers are... to review your work and let you know where you went wrong :)
I had thought I wil get to learn something after sharing my comments. But very sad it was not showed up here. At least wanted to know the reason..
@sadanand,
Your comment must have got missed. I get an email for every comment posted. I seem to have missed yours.
I would request you to re-post and I shall approve the same.
Okay.
A newbie working as a tester would hardly have any idea about how to actually think beyond the box. I am not saying test case writing helps to improve it. But she should be occasionally made to write them down to ensure to what extent she can ensure the test case coverage.
Maintaining test case/plan neither implies that we are going to promise quality nor does the exploration testing too.
The true value of plan/case etc documents comes into picture only when you happen to submit it as proof of what we have done and in case of something goingwrong at the end! Often for a long term project. Otherwise nobody even cares to see what it contains.
If "replace-able" commodity comes into pic, I would ask what kind of documentation did Lakshmipathi Balaji followed in cricket because of which he played really short for international cricket:-) what I mean is, these two fields are different and cannot be compared with each other. Family different from sports different from an organization:-)
I would say, we must have plan/case writing skill which should act as our strength in addition to exploratory testing.
If not a test case document, we must keep at least a brief checklist with us to ensure we are not missing out any obvious things.
@Sada,
100% correct. Just like how a newbie cricket player needs to be told:
"Step 1: Hold the bat
Step 2: Tap the bat a couple of times
Step 3: Wait for the ball to be released by the bowler
Step 4: Lift the bat and try to aim it at the incoming ball
Step 5: Hit the ball"
This world, with too much of bad testing has convinced people like you to believe that freshers would struggle without test cases.
Just check out this of how freshers behave when they are coached properly. Don't be surprised that their reports are far better than experienced ones: http://testertested.qualityfrog.com/erpstt.pdf
Right.
And thank you I will take "100% correct" as a reward assuming that you truly mean it.
@Sada,
Here is what I truly mean: I think you have got it wrong.
I understood pradeep,
All I meant wa, test case writing is a method which we can't totally rule it out and live without it.
I am on stressing on test case required only because we tend to forget what we do sometimes.
They are essential.
Yes I quickly read http://testertested.qualityfrog.com/erpstt.pdf
Yes the reports sound far better. Seems to be they are tendulkar and dravid of testing.
But again, PASSION AND ATTITUDE is what is exactly required despite of having everything in the background. Without those two factors it would be hard to survive, forget about test plan/case.
To add another point, whenever I said test case required, I did not mean that it is a low level test case describing in n number of steps. I meant, it should just highlight the scenario.
Why I said is, I again closely looked at the the example u gave about how a newbie taught about batting. This means you are assuming that I am telling we should stick to the test cases written in this pattern. I neither encourage that practice nor I follow it.
Nice post.More informative
Great Stuff.Thanks for the post
Hi Pradeep,
Totally agree with you regarding Test Cases and partially with Test Plan. I have couple of queries.
1. Yes you are right that 98%+ Test Plan are of no use, but as you are going to start working for a XYZ company for their really importance ABC project what is the mutual understanding or agreement between both the parties that these all the activities test team is going to execute.
If taking an example of House rent/sale agreement there are many clauses and other stuff mention in 3-4 pages of agreement that any of the party barely read or even the Notary/Advocate who has prepared. But still without that any deal won’t be consider as successed.
So it’s a business and one has to prepare a plan no matter how non-useful it is.
2. If your Product is 4 year old and there is some maintenance then how will you get the data of what all the things you had planned in the 3rd or 4th release of this product. Only test plan can tell you that without digging into more document (Here I am assuming that test team has updated test plan release wise).
3. Regarding Test Cases I agree with you that 2-3 step’s checks are enough to test a particular component/function instead of detailed complex test cases with lot of Pre-requisites, data set and blab la bla...
But the question here is I am not sachine or saurav ganguli nor I am an independent consultant but a Test Analyst with an elite xyz firm and I cannot leave this firm because of so n so reason. I hate writing detailed non-useable test scripts/cases (It also can be a long debate to differentiate test cases vs. test script) but my company want it because its their process. Now what the steps I should follow in this case?
Please at least give me reply for 3rd point I really looking for it.
Thanks,
Shailesh
The author writes both, sense and nonsense. Sense seems to come from the humongous literature that is freely dispensed over the world wide web. I am fan of his articles about testing in theory. The nonsense is the ill-processed output from his frustrated brain. His articles give strong vibes of him being a critic gone overboard.
Some of his articles such as the one above potentially damage the reputation Indian outsourcing industry. This one above, the author is clearly trying to cut out a sorry picture of the Indian testing industry when saying most of it's revenues come from writing test plans and test cases and reviewing them and so on. If that was the case his own enterprise would have been a trillion $ enterprise and the likes of Infy, TCS, Wipro etc. on the streets.
The author needs to know that testing is yet another revenue generating machinary of the corporate world. It's sheer business to rake in the moolah, he likes it or not. His overtly critical ideas of revolutionizing testing will not work in the real world.
@Anonymous,
You are right. The author is a critic gone overboard. He is such an asshole I can tell you that you would not want to dare reveal yourself in front of him. He is a bully too. Good that you hid yourself otherwise he would have bullied you. He is useless to the Indian testing and I have been telling this to people who thinks he is great.
On your comment of "his overly critical ideas of revolutionizing testing will not work in the real world" - you hit it on the nail. His ideas have never worked. Surprised that there are many stupid people who buy his ideas.
Hi Pradeep,
I have commented few weeks back on this post, my comment is just directly above this anonymous user's comment. I really want to know your view about the 3rd point i have asked. I guess it would be really helpful to me :)
Thanks,
Shailesh
@Shailesh,
Sorry for the delay in response but am glad you brought my attention back to what you want me to address. I didn't have time to open my own blog but today I have it. Thanks for your patience. I appreciate it 100%.
You are not Sachin of 2012 but do you see that you are how Sachin was in 1980? He practiced good cricket and that is the message he left to us.
There were situations in everybody's life that forces them to do something hard and cannot be done by everybody. Even for Sachin : http://www.ndtv.com/article/sports/how-sachin-tamed-temptations-to-become-a-legend-174797
So, what do you think you should be doing?
@Pradeep,
Thanks for the quick reply, and i realy glad to got this perfect answer :). I got what to be done in this scenario and will try to implement it accordingly.
Thanks
Hi Pradeep,
First off, I am relatively new in my learning of Context Driven testing, so pardon my possibly elementary question.
I do agree with the points that you mentioned, since I have personally witnessed Test Plans rot in the version-control systems in my previous companies.
In fact, I was able to convince management of my previous company to do away with the test plan documents.
However, I cannot seem to find a good argument to use about conducting regression testing without test case documents. Do you have any advice on this? Or maybe point me to articles that can help me further?
Thanks in advance
@ Ryan,
Before every flight, a pilot is supposed to check if everything that should be in place is in place. He does that through a checklist and not detailed scripts. Also, there are automated checks. So, a combo of automated checks and a human going through a checklist with exploration of risks for the change that was made can be of good help for regression. Thanks for asking.
Pradeep,
No matter how big a bully you are, like most bullies you seem to have more brawns than brains. I am glad, I could infuriate you with such sublte criticism.
I care not how big a bully you are and the fact that you started jumping the gun without even a little introspection highlights the fact that you lack patience, sense of constructive critism and the ability to accept negative feedback. All three most important atributes of a software tester.
@Anonymous,
I love the way you stress yourself.
Hi Pradeep,
Hope you will/had write/written a book on TESTING. Interesting Blog! Keep Blogging
Regards,
Nalini Thiyagarajan
Nice one Pradeep. This one is my fav "If all test case documents created so far were printed and burnt, we'd have fire for the next one thousand years."
I was so annoyed looking at test cases that I have put up a syntax for them at http://testtotester.blogspot.co.uk/2012/03/test-case-syntax.html Hope they get the sarcasm in it and not start following it ;)
@ Ryan - I don't want to hijack you to my blog from here but my last post was on regression testing (http://testtotester.blogspot.co.uk/2012/01/regression-checks-regression-testing.html) u might find it useful
The gist of your post is that test cases and test plans are a formality, but don't really amount to a pile of &*. Right? I have to agree with you mostly. They make up a pile of documentation that is used mostly for CYA purposes. However, there is some value in documentation to make sure the testing direction is correct, and that people are at least covering the minimum. We all know however that most defects are not found by fancy test cases and test plans, but by good testers who are familiar with the domain and application and have the freedom to do exploratory testing!
Post a Comment