So, the deal is to communicate the number when you know you are getting it right and buy time to get it right. By "right", I mean, more meaningful than you would have when you provided an estimate even without looking at the product.
I don't know what's wrong in going wrong the first time. Most of us learned to ride a bicycle, someone did it in a day and others took a month. How would you have answered a question, how long would you take to learn to ride a bicycle? Some people might need more time to get out of the fear of falling from it and others need time to get the balance right. It varies and hence cannot be determined before even someone got on to a bicycle?
Falsestimations and stress on teams
India being a hub for services, I often meet people who provided an estimate and struggle with it forever. For instance, a test manager provides an estimate to a client that it would take 5 working days with 5 testers to complete one full planned cycle of testing. When the test execution starts, the team discovers that the setup takes longer time than anticipated BUT in order to prove the estimation was right, the team puts in lots of extra hours and end up being stressed and frustrated. Stressed and frustrated people cannot produce good quality software. I think I don't need a PhD to say that.
That is exactly what is happening at an organization where my friend, colleague and one of the co-founder of Weekend Testing, Manoj Nair is working. He has just had a single Saturday & Sunday at home since he joined that organization 5 months ago and that too because he was unwell. Now, he is looking out for a job. What about the team's personal life and balance, who cares? Estimation is done and to be met.
Estimation of Estimation
How long do most people take to estimate? I could take a fraction of a second to say a number but in case I delay it by a couple of days and claim to use a model such as COCOMO (not DOCOMO) and then provide the same number, you might think it is meaningful.
Isn't this like the radio repairman trick?
In my childhood, television was a luxury and a radio was an essential. Sometimes when we took our radio to repair, the radio repairman would ask for 4 days time to get it fixed and charge us a good amount of money. The volume control doesn't work, 4 days of time to fix it. Have you known potentiometers? It wouldn't take 4 days even if his shop was flooded with radio sets to be repaired but it would have been hard to convince myself that I had to pay 40 rupees for a 5 rupee potentiometer and 5 mins of job to replace it with a new. So, the problem is with me as a customer and with people like me as customers.
If I let know the customer that he is wrong, he might not want to outsource the project to me, so I better say he is right. Simple business logic, ha?. How long are we going to offer that kind of service?
"Urge to talk" as a trap to effort estimation
In my experience, I have met a lot of managers who love meeting their teams because that is an opportunity for them to talk and for sure have listeners. I talk a lot, irrespective of being a manager or not but not at the cost of the delivery time unless the context demands to resolve an issue with the team. Turns out, I am just like you, who loves to talk.
However, some managers I have worked with talk so much that the meeting drifts from the agenda and goes to Alice in Wonderland, Perils of Penolope Pitstop, and stories of Shrek from Part 1 to Parts that are yet to be planned. For instance, I once had a 4 hour meeting with my manager who asked me to explain why I missed finding a specific bug in the feature I was testing. How dumb!
So, shouldn't such managers include the time they are going to waste of the team as a part of their estimation?
What if that is included? The manager is sacked. Should I as a manager do something knowing that I'd be sacked. Oh no, I am not dumb, I am a smart ass, let me not do that. Instead let the team take the blame. What if the team opposed such a thing happening? Why would they? I control their appraisals and hike.
How do I estimate effort? ( In other words most meaningless and impractical test effort estimation)
If you were my client and asked me "How long would your team take to test?", I am curious about what problems am I trying to solve for you in a cycle of testing. That is important and just as important as the need to provide an estimate. I have a bunch of other questions to ask you that are relevant to knowing what kind of information you want and at what frequency.
Then comes Q-Pra-Deep ( Q-Pra-Deep is derived from Rapid Software Testers)
Then, I would now need your permission to go wrong at least thrice with my estimation before I be of more value to you. I know what you are thinking, that is why I said this approach would be the most meaningless and impractical way to test effort estimation.
Quick: Now, I'd do the same as other people and spit out a number but to arrive at this number, I'd use my experience. The difference here is, I wouldn't take a lot of time to arrive at this number. I provide this number quick. What if I don't have experience in it, poll the network and seek the advice to arrive at a number quickly.
The communication that I'd do to the client is that at this stage is that the number I provided would be at most 50% incorrect.
Practical: After providing a quick answer, I'd run a cycle or two of the work we discussed and learn
- How long the work is actually taking?
- Make conscious efforts of understanding the ground reality as opposed to my assumptions.
- Probe, Interview and Quiz team members about their experiences.
- Keep a constant tab on the mission versus achieved result.
- Make notes of areas that the team needs to upgrade in order to deliver faster.
At the end of the cycle of testing and analysis of the deliverable versus the mission, I'd have a number that is likely to be different than the number that I provided during the Quick approach. This is a point to communicate the second number to the clients to help them plan and budget the future work. Oh, I said it twice. This is for sure impractical. No client would entertain that. They want us to be right the first time just because they walked the first time they tried doing it when they were a child.
Let me clarify when I say "Practical", it is to me and not necessarily you.
Deep: As I run a couple of cycles with my Practical approach outcome number, I'd have tremendous learning of things that interrupt me in making the Practical, practical. So, I want to pay attention to each of them and provide you with a deep analysis of estimation. For instance, how long would my team take to test for a newly available build while they are already mid-way testing a build?
- Not interesting? While it looks like the same number as the Practical number should be applicable, how about the time of uninstalling the build, ghosting the machines if needed, re-installing the components, cleaning up the registry, clearing the database, hey how about taking a break before starting it all over again?
- What about contexts where the client demands a demo release to be given to their customers while we are somewhere, somewhere near to completing a cycle. How long would it take to test the demoable version of the software?
- Ah! There are non-bug-finding-tasks such as supporting developers after reporting a few bugs, participating in bug triage meetings, managing leaves of team members, organization events such as annual day, lunch out sessions.
Would I stick to the number I provided?
Doesn't it depend on the mission and context? My core aim is to answer questions that the management and customer has about the product my team and I were testing it.
So, Quick is quick, Practical is practical and Deep depends on how deep you'd like to go. Oh, you plan to hire me and read this post. Sorry, I can do other kinds of estimation as well.
Have I tried this? Oh yes, and my client was my reporting manager at a product company that I worked as a test manager.
Wasn't I opposed to do it this way?
Of course, yes. Why on earth would anyone allow such a meaningless and impractical approach to test estimation happen. So, here is my experience report:
Before I was hired, Dev Manager (also partly Product manager) who was handed over the task to manage test teams. Not just his problem, he appeared more slant to the Dev than test. He did a good job of Dev although it wasn't convincing to me about the test part. By running fewer tests and in a short span of time, the bugs found by the test team weren't looking great. However, this made Dev looked robust. Although it wasn't supposed to be seen that way that is what happened.
When I took over, one of my mission was to help test teams find more bugs before customers finds many of them. In the process of trying to achieve the mission I found problems with the estimation that had happened and the trap of sticking to the numbers provided to the management.
So, as a first attempt, I tried seeking his consultation to estimate. He didn't appear to like these ideas (just like how you might feeling after reading this) and then I agreed to run one cycle of testing with his estimate provided but with a check on mission and the tests defined by me. It didn't work out and he too agreed that. The number of important problems or bugs (although it ain't good way) I found supported me that different tests had to be run and hence the estimate wouldn't match.
So, I tried asking for 2 attempts to re-estimate. I wasn't permitted. Luckily he got too busy with other product issues and I took the liberty to re-estimate. Although they weren't approved, turned out that the different tests I helped the team create, really took as much time as I had estimated. So, I had a good case to present to them and myself. I think the focus shifted from "Is Pradeep's estimate meaningful?" to "How do we fix 600 major issues?"
Thanks to Ewald Roodenrijs for asking me questions about test effort estimation over wave that made me to decide to write a post on it. Also thanks to my brother Michael Bolton for quizzing me how I'd do estimation over Skype a couple of months back that help me write down my ideas.
Inspirations, References, & Suggested Reading:
- A statement I learned and practiced to say from Michael, goes something like, "I want to help you but I don't know how to help you. I can be of help to you provided you help me with this."
- Test Project Estimation - The Rapid Way - Michael Bolton
- Test Trimming - A fable about Testing - Jerry Weinberg
- Traps in Test Estimation - Shrini Kulkarni
- Rapid Software Testing - James Bach & Michael Bolton
- Test Estimation is Really Negotiation - The same Michael Bolton
- Testing the Limits with James Bach (Part1) - James Bach
Note to all organizations who outsource their testing work : If you are considering to outsource your testing work anywhere in this world and want the service provider to provide right estimates at the first attempt, you aren't outsourcing ready, yet. Well, however, we (service providers) don't mind receiving your business. It's our pleasure to serve you in the way you want it to be.