"Some birds aren't meant to be caged, their feathers are just too bright"- Morgan Freeman, Shawshank Redemption. This blog is from one such bird who couldn't be caged by organizations who mandate scripted software testing. Pradeep Soundararajan welcomes you to this blog and wishes you a good time here and even otherwise.

Wednesday, April 28, 2010

Test coverage : Life beyond functionality

At a class recently 

Scene 1:

"What have you all been doing for the last few years in this organization?"

Chorus: "We test for functionality of the software?"

"That's nice! Why do you think it is called /functionality/?"


"OK, what is a function?"


"No problem. Let's fix it. Where else have you heard the word /function/? Math? So we all might have studied f(x) = y and that could have been our introduction to working with functions. Later we moved on to parabolas and hyperbolas which had functions of different behavior and complexity. So what we are doing when we say /functional testing/ is, we provide inputs X to a function f(X) and monitor Y. We claim to know that f(X) should be Y and we also claim to know the range of Y. So, that forms our oracle to find bugs with functional testing. /Y/ doesn't necessarily need to be a numeric value in functional software testing".

Scene 2:

"So, what else do you test beyond functionality?"
  • "Our bug reports get rejected if we report non functional bugs so we don't test for it", 
  • "There is a team in US who is supposed to do it"
  • "When we find a non functional bug and report it, we are asked why did we spend time deviating from the scope given to us"
  • "We have enough test cases in functional testing that we don't have time for other kinds of testing"
"Are you doing Quality Assurance or Software Testing?"


"Let me explain. In Software Testing, you provide quality related information to help the management take better informed decisions. I think in Quality Assurance, you claim to provide confidence to the management that you have checked and tested a lot of things (not just software) and things didn't change after your checks and hence its OK to use the word /assure/ with another word /quality/."
  • "I think we do a mix of both"
  • "Our designation is Software Tester but we are internally called QA"
  • "I don't know. I am doing my job and earning my paycheck"
  • "Yikes, this is confusing"
  • "I thought Testing == QA" 
 Scene 3:

 "I appreciate the diversity. Tying it back to our previous topic, irrespective of what you do (Functional Testing or Functional QA, the information you provide to your management is so weak as compared to what you were supposed to be doing."

"How is that?"

"Test coverage is the extent to which we have modeled and tested the system -- Michael Bolton. You appear to have modeled the system for functionality and you could model your system in more ways within and beyond the scope of functionality to learn more about the product and to be of more value to your management. Assuming you are interested to do that. Here is a list of thing that you might want to focus on:

Product Elements Coverage
  • Data
  • Platforms
  • Operations
  • Time
  • Structure
  • Functions
Quality Criteria Coverage
  • Capability
  • Reliability
  • Usability
  • Scalability
  • Performance
  • Installability
  • Compatibility
  • Supportability
  • Testability
  • Maintainability
  • Portability
  • Localizability
Technique Coverage
  • Scenario based
  • Claims based
  • User based
  • Flow based
( refer to Rapid Software Testing slides for more )

"That looks like a lot of thing to do and I know these are important for quality. We don't have time for functionality and how can we think about all these things?"

That's a nice thing that you realize you don't have time even for functional testing to be done. How about asking the management to re design and re think about the time that is given to you and the value you can deliver?

Scene 4:

"I know for sure. They won't allow us to do all this"

"Interesting. Is that your assumption or a fact?"

"That's what is going to happen if we ask and we know that."

"Have you tried it?"

"No, but we know them very well."

"How much do you think they know about you and your skills?"

"They hardly know about me and my skills."

"So do you about them. If they assume you are not skilled to do other kinds of testing and also assume you heard them say /We are sure, they don't know to test beyond functionality/, how would you react to it?"

"I would show to them that I can"

"How about giving them a chance to show that they too can change?"


Two things:

  • Give each of your colleague a chance, explain to them your problem.
  • Give yourself a chance, focus on your test coverage and explore a life beyond functionality.

Friday, April 16, 2010

Experience Report : BWST 2

 Experience Report - Bangalore Workshop on Software Testing - 2
3rd April, 2010 @ Shilton Royal, Bangalore

Bangalore Workshop on Software Testing stepped into its 2nd year on April 3rd, 2010. Why wouldn't it?
This time, it was much more organized than BWST 1. Of course, we learn some lessons, if not all, when we do something.

The groundwork required announcing it on my blog, managing the registrations, speaker invites, cancellations, getting sponsorship, managing the budget, conference and hotel booking. As we had Selim Mia from Bangladesh as a participant, the work also involved helping Selim getting a Visa to India. Santhosh and Parimala were of good help for me in the ground work.

So, the D-day arrived. Some new faces and some old started dropping a couple of minutes earlier to the start time published to them. What? An Indian conference happening on time? That's crazy ain't it? So, till the time arrived some participants started to discuss about Left, Right, Neutral of Politics, Democracy, India and more.

We started off at 9 AM with a check in & self introduction from all participants and Parimala explaining the rules of the day and how to use K cards. She announced herself as Kaali Maata for the day who would facilitate the workshop.

Sharath Byregowda's talk on "How he cleared the trap that prevented him from attending BWST2"

What better way to start than to get a guy who was not likely to attend BWST2 because he had to be in office for someone's estimation being the biggest ever blunder committed. Sharath talked how he cleared the trap to be able to attend BWST2. While he was talking, lots of green cards started popping up. He explained how bugs (that were show stoppers) allowed the test team to buy some time while the development team were fixing them. 

Rahul Verma raised a question, "How can you go and look for show stoppers? Isn't it only when you find a bug you'd know if it is a show stopper?". That triggered some red, blue and green cards to show up. The discussion shifted from that to Severity, Priority and flowed into certifications. Sharath talked about the experience he had in proposing BBST course in his organization.

Ashok's talk on Metrics

So, Ashok T, CEO of Stag Software brought in his wide experience of dealing with metrics and questioned what metrics made sense. He talked about metrics used in various stages of software development and testing and shared his experiences of working with several clients on them. He talked about having helped his clients understand that collecting all these metrics is good but do we really know why we are doing it and what is our goal?

Discussions evolved around metrics and estimation. Not in anyone's experience there, an estimation had gone right. Rahul Mirakhur had to say, "If you get the estimate right, then its not right". There appeared to be a lot of interest from most people to try to get estimation right. I don't think anyone should try getting their estimation right because that is not the only thing that would be helpful to the clients they work for.

Selim Mia

We had a participant and speaker from Bangladesh, Selim Mia. Such passion is great to see and experience. He traveled by train from Bangladesh to Bangalore and back. The moment he stood up, everyone asked him a question unanimously : Please tell us how is testing done in Bangladesh? He had to answer that before he could proceed on his planned talk. I personally think there is lot of potential in Bangladesh but I also hope they avoid falling to traps that we have fallen into.

Selim had a combo for us : An experience report and sought answers to questions he had as a test manager. Based on what he said it occurred to me that the senior management there at Bangladesh were in for results. So, sometimes they gave enough freedom to testers to achieve results - which is good.

Discussions returned to estimation, metrics, scripted and exploratory testing. Vipul had been to Bangladesh, so he shared his experience of interacting with people there and termed it "pleasant". Two names that Indians now know from Bangladesh is "Sajjadul Hakim" and "Selim Mia". I think its time we know more.


BWST 1 taught us a few lessons. We went out for lunch and in search of a hotel and spent about 2 hours in BWST 1 on it. This time we organized it in the same hotel and hence we saved a lot of time. We spent munching a good lunch for 50 minutes and then got back to action. Good lunch, I liked the soup & noodles!

Rahul Verma

I was invited to a conference in Germany and I couldn't make it. Instead of me, Rahul Verma did and I think it was a good thing to have happened for the conference. I believe Rahul Verma can deliver the blows that audience needs and that is exactly what he did. His presentation was an exploration into what crap, trap and good means to you versus to those whom you report to or to who reports to you. He delivered some punches to bloggers like me, which I welcome :)

He shared a few experience reports from his office and then led to asking us if we do have an opinion about what we say and how did we arrive at it? He left with a message saying, "Evaluate"

Those who probably did not know Rahul Verma were probably pleasantly shocked by his presentation and I argued over having an opinion and not having an opinion. I feel opinions are in making. Not expressing opinions could be dangerous in some situations while expressing it could be dangerous in a few others.

Vipul Kocher

Testers break rules. Good testers know when it is safe to break rules. So did Vipul. He cut the crap about  crap in the theme and focused on helping us learn the Noun and Verb technique of generating test ideas. The first thing that strikes is his acknowledgment to Elizabeth Hendrickson. It is a strong message to the testing community to owe credits to someone who has helped an idea to be helpful to a larger mass of testing community. Although "Noun and Verb" technique was unheard to many, they did enjoy learning it from someone who had implemented it in the past and shall continue to work on it. I usually don't run out of test ideas and I think through his presentation, the likeliness of me running out of test ideas has further reduced.

Ashok had questions to ask about Noun and Verb technique and I found them interesting. I too have plenty of questions about it but I am going to be working on it for a while before I try to find out answers. There were a few junior level testers who were at BWST this time such as Sai Divya & Shwetha Ghorpade who acknowledged that it will be interesting to implement the technique at their work.

Meeta Prakash

Meeta hit the bell. She took the word crap from the theme and thought about one of the crappiest thing at work - meeting. Her session didnt need any facilitation. She kept polling the audience and questioning things about meeting. It was interesting to know every person except Allmas (lucky Indian) thought there was a lot of time being wasted in meeting. Sometimes people not needed are pulled into a meeting or other times people who are most needed in the meeting are left out because they speak truth. I got reminded of a meeting money burn meter that I saw sometime back which shows how many dollars are burnt in the meeting.

The discussion was interesting. I was reminded of one my ex-manager whose meeting only gets over when his wife calls him. Those who reported to him felt his wife was a savior and a viking.People brought in their experiences of craps and traps of meeting. I once worked for a CMM Level 3 company who was trying hard to achieve CMM Level 5 status. An SEPG team in there ( don't know SEPG : Software Engineering Process Group ) wasted a lot of our testing time. SEPG team ensures that a spelling mistake bug is as expensive as a database corruption by involving the whole team to do a root cause analysis of how the spelling mistake bug went unnoticed. Vasu brought in his experience of how he learned to handle customers diligently  by observing his manager handle it in a meeting. Allmas is way too lucky. She appears to have a team that handles meeting so well. Don't envy her, she is going to be as unlucky as you when she moves out of the organization.

Sukanta Bhatt

Oh this man! The stories he shared on testing medical devices got people doing two things - laughing while thinking. People didn't seem to want him stop. He shared experiences that led to discover new things and realize the value of interacting with customers and being at the customers place watching them how they use the product we develop and test. Was a cool way to end BWST 2 presentations.

Jantha (participants) got curious about medical applications and devices testing space and started to pound him with questions and asking more experience reports on medical devices. Well, that happened over a beer.


So, when the bell rang at 5:30 PM, we officially concluded BWST 2. Before we did that, we had a checkout in which each person given less than a minute were asked to talk about one take-away from the whole day. Rahul Verma and Sharath believe that it was cruel of me to ask for "one" takeaway while there were lots. Not that others had just one but these people voiced their opinions about it. The flip side is if someone who is new to this concept of BWST and had been all silent, is easy for them to talk about one take way unlike others who are practiced to speaking a lot.

Ashok & Vipul

The audience weren't exhausted. So, they ignored my call for heading to a pub nearby and asked Ashok and Vipul to talk about experience of running a testing services organization. Questions were posed about China, US, Europe and ANZ regions. The thrust was how do we as Indian testers do better.


So, we all got shot at the end by one of the hotel staff :)

Beer in Pub

We decided to catch up in Enigma, the Pub @ Koramangala and testers rocked the place. A couple of pitchers and soft drink went in. Girls who had been to BWST 2 also accompanied us to a pub as they knew they are hanging out with some of the gentlemen of the industry. That's it.

Note of thanks

To all participants : Santhosh Tuppad, Sharath Byregowda, Selim Mia, Ashok T, Rahul Verma, Vipul Kocher, Dr.Meeta Prakash, Sukantha Bhat, Swetha Ghorpade, Senthilnathan, Allmas Mullah, Eshwar Kumar, Lakshmi Narasimha, Gokul, Dhanasekar S, Rayanagouda Patil, Vasu Swaminathan, Rahul Mirakhur, Mandeep Singh, Ajay Balamurugadas, Ravisuriya, Yeshwanth Rao, Sai Divya, Chandrasekha, Parimala

To our dear event sponsor: Vipul Kocher
To my co-organizer, Santhosh Tuppad
To the facilitator, Parimala
To you, for reading this.

I shall see at least some of you at BWST 3!