So, the deal is to communicate the number when you know you are getting it right and buy time to get it right. By "right", I mean, more meaningful than you would have when you provided an estimate even without looking at the product.
I don't know what's wrong in going wrong the first time. Most of us learned to ride a bicycle, someone did it in a day and others took a month. How would you have answered a question, how long would you take to learn to ride a bicycle? Some people might need more time to get out of the fear of falling from it and others need time to get the balance right. It varies and hence cannot be determined before even someone got on to a bicycle?
Falsestimations and stress on teams
India being a hub for services, I often meet people who provided an estimate and struggle with it forever. For instance, a test manager provides an estimate to a client that it would take 5 working days with 5 testers to complete one full planned cycle of testing. When the test execution starts, the team discovers that the setup takes longer time than anticipated BUT in order to prove the estimation was right, the team puts in lots of extra hours and end up being stressed and frustrated. Stressed and frustrated people cannot produce good quality software. I think I don't need a PhD to say that.
That is exactly what is happening at an organization where my friend, colleague and one of the co-founder of Weekend Testing, Manoj Nair is working. He has just had a single Saturday & Sunday at home since he joined that organization 5 months ago and that too because he was unwell. Now, he is looking out for a job. What about the team's personal life and balance, who cares? Estimation is done and to be met.
Estimation of Estimation
How long do most people take to estimate? I could take a fraction of a second to say a number but in case I delay it by a couple of days and claim to use a model such as COCOMO (not DOCOMO) and then provide the same number, you might think it is meaningful.
Isn't this like the radio repairman trick?
In my childhood, television was a luxury and a radio was an essential. Sometimes when we took our radio to repair, the radio repairman would ask for 4 days time to get it fixed and charge us a good amount of money. The volume control doesn't work, 4 days of time to fix it. Have you known potentiometers? It wouldn't take 4 days even if his shop was flooded with radio sets to be repaired but it would have been hard to convince myself that I had to pay 40 rupees for a 5 rupee potentiometer and 5 mins of job to replace it with a new. So, the problem is with me as a customer and with people like me as customers.
If I let know the customer that he is wrong, he might not want to outsource the project to me, so I better say he is right. Simple business logic, ha?. How long are we going to offer that kind of service?
"Urge to talk" as a trap to effort estimation
In my experience, I have met a lot of managers who love meeting their teams because that is an opportunity for them to talk and for sure have listeners. I talk a lot, irrespective of being a manager or not but not at the cost of the delivery time unless the context demands to resolve an issue with the team. Turns out, I am just like you, who loves to talk.
However, some managers I have worked with talk so much that the meeting drifts from the agenda and goes to Alice in Wonderland, Perils of Penolope Pitstop, and stories of Shrek from Part 1 to Parts that are yet to be planned. For instance, I once had a 4 hour meeting with my manager who asked me to explain why I missed finding a specific bug in the feature I was testing. How dumb!
So, shouldn't such managers include the time they are going to waste of the team as a part of their estimation?
What if that is included? The manager is sacked. Should I as a manager do something knowing that I'd be sacked. Oh no, I am not dumb, I am a smart ass, let me not do that. Instead let the team take the blame. What if the team opposed such a thing happening? Why would they? I control their appraisals and hike.
How do I estimate effort? ( In other words most meaningless and impractical test effort estimation)
If you were my client and asked me "How long would your team take to test?", I am curious about what problems am I trying to solve for you in a cycle of testing. That is important and just as important as the need to provide an estimate. I have a bunch of other questions to ask you that are relevant to knowing what kind of information you want and at what frequency.
Then comes Q-Pra-Deep ( Q-Pra-Deep is derived from Rapid Software Testers)
Then, I would now need your permission to go wrong at least thrice with my estimation before I be of more value to you. I know what you are thinking, that is why I said this approach would be the most meaningless and impractical way to test effort estimation.
Quick: Now, I'd do the same as other people and spit out a number but to arrive at this number, I'd use my experience. The difference here is, I wouldn't take a lot of time to arrive at this number. I provide this number quick. What if I don't have experience in it, poll the network and seek the advice to arrive at a number quickly.
The communication that I'd do to the client is that at this stage is that the number I provided would be at most 50% incorrect.
Practical: After providing a quick answer, I'd run a cycle or two of the work we discussed and learn
- How long the work is actually taking?
- Make conscious efforts of understanding the ground reality as opposed to my assumptions.
- Probe, Interview and Quiz team members about their experiences.
- Keep a constant tab on the mission versus achieved result.
- Make notes of areas that the team needs to upgrade in order to deliver faster.
At the end of the cycle of testing and analysis of the deliverable versus the mission, I'd have a number that is likely to be different than the number that I provided during the Quick approach. This is a point to communicate the second number to the clients to help them plan and budget the future work. Oh, I said it twice. This is for sure impractical. No client would entertain that. They want us to be right the first time just because they walked the first time they tried doing it when they were a child.
Let me clarify when I say "Practical", it is to me and not necessarily you.
Deep: As I run a couple of cycles with my Practical approach outcome number, I'd have tremendous learning of things that interrupt me in making the Practical, practical. So, I want to pay attention to each of them and provide you with a deep analysis of estimation. For instance, how long would my team take to test for a newly available build while they are already mid-way testing a build?
- Not interesting? While it looks like the same number as the Practical number should be applicable, how about the time of uninstalling the build, ghosting the machines if needed, re-installing the components, cleaning up the registry, clearing the database, hey how about taking a break before starting it all over again?
- What about contexts where the client demands a demo release to be given to their customers while we are somewhere, somewhere near to completing a cycle. How long would it take to test the demoable version of the software?
- Ah! There are non-bug-finding-tasks such as supporting developers after reporting a few bugs, participating in bug triage meetings, managing leaves of team members, organization events such as annual day, lunch out sessions.
Would I stick to the number I provided?
Doesn't it depend on the mission and context? My core aim is to answer questions that the management and customer has about the product my team and I were testing it.
So, Quick is quick, Practical is practical and Deep depends on how deep you'd like to go. Oh, you plan to hire me and read this post. Sorry, I can do other kinds of estimation as well.
Have I tried this? Oh yes, and my client was my reporting manager at a product company that I worked as a test manager.
Wasn't I opposed to do it this way?
Of course, yes. Why on earth would anyone allow such a meaningless and impractical approach to test estimation happen. So, here is my experience report:
Before I was hired, Dev Manager (also partly Product manager) who was handed over the task to manage test teams. Not just his problem, he appeared more slant to the Dev than test. He did a good job of Dev although it wasn't convincing to me about the test part. By running fewer tests and in a short span of time, the bugs found by the test team weren't looking great. However, this made Dev looked robust. Although it wasn't supposed to be seen that way that is what happened.
When I took over, one of my mission was to help test teams find more bugs before customers finds many of them. In the process of trying to achieve the mission I found problems with the estimation that had happened and the trap of sticking to the numbers provided to the management.
So, as a first attempt, I tried seeking his consultation to estimate. He didn't appear to like these ideas (just like how you might feeling after reading this) and then I agreed to run one cycle of testing with his estimate provided but with a check on mission and the tests defined by me. It didn't work out and he too agreed that. The number of important problems or bugs (although it ain't good way) I found supported me that different tests had to be run and hence the estimate wouldn't match.
So, I tried asking for 2 attempts to re-estimate. I wasn't permitted. Luckily he got too busy with other product issues and I took the liberty to re-estimate. Although they weren't approved, turned out that the different tests I helped the team create, really took as much time as I had estimated. So, I had a good case to present to them and myself. I think the focus shifted from "Is Pradeep's estimate meaningful?" to "How do we fix 600 major issues?"
That's it!
Thanks to Ewald Roodenrijs for asking me questions about test effort estimation over wave that made me to decide to write a post on it. Also thanks to my brother Michael Bolton for quizzing me how I'd do estimation over Skype a couple of months back that help me write down my ideas.
Inspirations, References, & Suggested Reading:
- A statement I learned and practiced to say from Michael, goes something like, "I want to help you but I don't know how to help you. I can be of help to you provided you help me with this."
- Test Project Estimation - The Rapid Way - Michael Bolton
- Test Trimming - A fable about Testing - Jerry Weinberg
- Traps in Test Estimation - Shrini Kulkarni
- Rapid Software Testing - James Bach & Michael Bolton
- Test Estimation is Really Negotiation - The same Michael Bolton
- Testing the Limits with James Bach (Part1) - James Bach
Note to all organizations who outsource their testing work : If you are considering to outsource your testing work anywhere in this world and want the service provider to provide right estimates at the first attempt, you aren't outsourcing ready, yet. Well, however, we (service providers) don't mind receiving your business. It's our pleasure to serve you in the way you want it to be.
10 comments:
The bicycle analogy reminds me of my tryst with the bicycle. I took about a week to ride the bicycle my dad bought for me. My dad was convinced that I was ready to ride all alone. I just rode 1/4 km when a 2 yr old child threw himself onto my bicycle. I was in a state of trance for the next few seconds. The front wheel has gone over the child's body, and fortunately, I had stopped the second wheel in a fraction of a second. All the child did after this incident was to run towards his friends instead of crying. I was looking for his mom, but looks like she was busy in the kitchen while the kid enjoyed being on the road :-).
At my current company, we follow the Scrum methodology. It's been 20 sprints so far and yet we fail at estimating correctly. Why do most of us fail in estimation? Often times, we(both programmers and testers) are given 4-6 hrs time to estimate about the tasks that would last an entire sprint? Impractical and illogical. Programmers often fail to provide correct estimates because they don't know the challenges in the underlying tasks (unless picked one by one and studied in detail which would need more time). Testers often don't consider customer escalations and high priority testing activities which boomerang time and again. Immediate reporting managers are aware of these problems, yet they fear their managers who in turn fear their managers and so on. Nobody wants to speak up. Perhaps, nobody with a red flag is encouraged. All they want is GREEN. Nobody wants to ask 'Is there a problem here'. We are busy asking 'Are we done?'
I don't blame the management alone. It's each one's responsibility in the project to educate every person including ourselves about the day to day challenges involved in estimation. There is a bigger problem here - Are you encouraged to speak up?
Lack of communication is the root cause of many problems existing in this world today (I think so :-))
Q-Pra-Deep is something that I need to learn about in detail as well. Thanks for pointing to this.
Very good post,
Parimala Shankaraiah
Less estimations will result in less traps
One advice which is true whenevr your estimation is in the Quick, practical or in deep level - try to leave items which are not under your control, but has impact on your actual schedule - outside the estimation.
One example is when defining Entry criterion together with the estimation, so to clarify that the estimation given depends on having a code which is testable
( and defining what exactly means that the code is testable :-) )
There are some key things to consider when esitmating.
Managers should only provide a ballpark number which should be subjected to change based on the individual contributor about to test the product.
Every individual tester should be accounted for 70-80% of their time for testing because of random stuff like consulting, process, documentation, etc.
Every test team must manage a chart listing down the tasks big or small and should be accounted for estimates. Its very important like I usually forget to account for designing better tests so that I can come up with utils like special purpose decoders or something similar.
Efforts must always consider what level of testing was performed parallel to development as this can change numbers significantly.
last not least... when working as service team and a project is outsourced to you, always request an estimation expected from the customer and then present a reasonable case to justify your numbers. It is also ball parking since they might give you a smaller estimation and you might feel uncomfortable going beyond them but helps you in being reasonable.
On different projects for different reasons I have used (rather say, attempted to use) all the techniques that are discussed about: Gut feeling (ie experience :) ) , TCP and FP
Every time when I have been asked to provide estimation, I have always had one question: How in the world are we trying to find out how much time we will take to do a job/activity about which
• We don’t know the definite size
• We don’t know the definite complexities interms technology involved in
• We don’t know the definite set of other software that it interfaces with and their complexities and influences
• We don’t know the definite set of scenarios in which the software is going to be consumed
• We don’t know the definite set of functional complexities (I wonder if there is)
The list (of things that we don’t know) potentially grows.
In my opinion, we can perhaps look at how much time I have with me rolling out a product with how much money (budget).
may be this bottom up approach would work then playing smart with uncertain yet influential parameters
A wonderful and most important post Pradeep.Most of the time the Estimations are just as per Manager's own calculations which are to please the Higher management to show "How good they are in making people work" rather than actually how much time is necessary,I feel most of them I least bothered in Testing or the team working under them, they are their slaves helping them to get their Salaries,incentives,awards and good Appraisal's.
Cheers,
Jassi
You can never say that your estimation is going to be under any circumstances. Always there will be some assumptions which are mutually understood by the client and service provider.
In My existing project,I have been working as test manager for the last 3 years and most of the times test estimates are proper. I know why i failed in some cases.
When I give estimation
I consider the following
1)Am i having commendable knowledge in that particular function?
2)Is it existing functionality? If yes refer old test specs to get some idea and cross check yourself
If NOT then think about all possible which you can think of
3) I do consider Demos to customer,Environmental problems (Based on past history), probable defect testing cost,Build, Any environmental setup requirement, Training to team members in the team, Test document reviews . Each activity that we do
4)I am sure that any estimation in fact would be done based on some assumptions and some probable contingency factors.
I like your many of posts and the way it thinks. frankly speaking
I don't find this article as so useful and thought provoking one. I am always happy to argue with people who thinks deeper and deeper. Because such arguments would make us to think more and more.
Eagerly waiting to see how your reply would be. I would always come to you with a negative mind. If I turn out to be positive at the end then there is a lesson that i need to learn
Pradeep,
Thanks for your information. I'm going to use this in the future. I wrote a post on my blog about how I estimate the testing. It's mostly based on checking.
http://www.testingthefuture.net/2010/03/using-metrics-to-estimate-your-testing/
Hi Pradeep,
I think that software projects are probably the hardest projects to estimate, and that's why most project managers are very far from their initial estimates by the end of the project.
I would like to republish your post on PM Hut where many project managers will be able to be benefit from it. Please either email me or contact me through the "contact us" form on the PM Hut website in case you're OK with this.
Hi Pradeep,
I know this is an old post, but i am currently at a point where i need to provide test effort estimation.
I have read many articles that go from simple to complex, but i was hoping that you might be able to provide some practical example of test effort estimation from your experience.
If for example i have a Requirement spec and the engineer has provided a dev estimate, i now need to provide a QA estimate.
Could you please describe what your approach would be?
Thanks,
@Irfaan,
You seem to not really be reading and learning but skimming through. I had answers to your question in my post or in the links I provided.
Post a Comment