Scene 1:
"What have you all been doing for the last few years in this organization?"
Chorus: "We test for functionality of the software?"
"That's nice! Why do you think it is called /functionality/?"
Silence
"OK, what is a function?"
Silence
"No problem. Let's fix it. Where else have you heard the word /function/? Math? So we all might have studied f(x) = y and that could have been our introduction to working with functions. Later we moved on to parabolas and hyperbolas which had functions of different behavior and complexity. So what we are doing when we say /functional testing/ is, we provide inputs X to a function f(X) and monitor Y. We claim to know that f(X) should be Y and we also claim to know the range of Y. So, that forms our oracle to find bugs with functional testing. /Y/ doesn't necessarily need to be a numeric value in functional software testing".
Scene 2:
"So, what else do you test beyond functionality?"
- "Our bug reports get rejected if we report non functional bugs so we don't test for it",
- "There is a team in US who is supposed to do it"
- "When we find a non functional bug and report it, we are asked why did we spend time deviating from the scope given to us"
- "We have enough test cases in functional testing that we don't have time for other kinds of testing"
"What?"
"Let me explain. In Software Testing, you provide quality related information to help the management take better informed decisions. I think in Quality Assurance, you claim to provide confidence to the management that you have checked and tested a lot of things (not just software) and things didn't change after your checks and hence its OK to use the word /assure/ with another word /quality/."
- "I think we do a mix of both"
- "Our designation is Software Tester but we are internally called QA"
- "I don't know. I am doing my job and earning my paycheck"
- "Yikes, this is confusing"
- "I thought Testing == QA"
"I appreciate the diversity. Tying it back to our previous topic, irrespective of what you do (Functional Testing or Functional QA, the information you provide to your management is so weak as compared to what you were supposed to be doing."
"How is that?"
"Test coverage is the extent to which we have modeled and tested the system -- Michael Bolton. You appear to have modeled the system for functionality and you could model your system in more ways within and beyond the scope of functionality to learn more about the product and to be of more value to your management. Assuming you are interested to do that. Here is a list of thing that you might want to focus on:
Product Elements Coverage
- Data
- Platforms
- Operations
- Time
- Structure
- Functions
- Capability
- Reliability
- Usability
- Scalability
- Performance
- Installability
- Compatibility
- Supportability
- Testability
- Maintainability
- Portability
- Localizability
- Scenario based
- Claims based
- User based
- Flow based
"That looks like a lot of thing to do and I know these are important for quality. We don't have time for functionality and how can we think about all these things?"
That's a nice thing that you realize you don't have time even for functional testing to be done. How about asking the management to re design and re think about the time that is given to you and the value you can deliver?
Scene 4:
"I know for sure. They won't allow us to do all this"
"Interesting. Is that your assumption or a fact?"
"That's what is going to happen if we ask and we know that."
"Have you tried it?"
"No, but we know them very well."
"How much do you think they know about you and your skills?"
"They hardly know about me and my skills."
"So do you about them. If they assume you are not skilled to do other kinds of testing and also assume you heard them say /We are sure, they don't know to test beyond functionality/, how would you react to it?"
"I would show to them that I can"
"How about giving them a chance to show that they too can change?"
Silence
Two things:
- Give each of your colleague a chance, explain to them your problem.
- Give yourself a chance, focus on your test coverage and explore a life beyond functionality.
"Give yourself a chance, focus on your test coverage and explore a life beyond functionality."
ReplyDeletePradeep - I Agreed with you.
(But test coverage doesn't mean that if scope is functional testing, then how non-functional testing is included in test coverage? anyways, we can ignore this question for now)
My Story -
Most of the times we don't have time to explore life beyond functionality.
We are in Products and it is a enterprise level product. Client uses the software as per given directions.
Also I am agree with the following points -
"When we find a non functional bug and report it, we are asked why did we spend time deviating from the scope given to us"
"Our bug reports get rejected if we report non functional bugs so we don't test for it".
Above two points are true and happens in many companies.
My Assumption -
But yes, if we start exploring life beyond functionality, then it will help us (testers) in thinking out of the box and increase lateral thinking skills, which indirectly helps in doing better functional testing.
@ OUG,
ReplyDeleteBut test coverage doesn't mean that if scope is functional testing, then how non-functional testing is included in test coverage? anyways, we can ignore this question for now
A data coverage is meaningful within the scope of functional testing.
Various non functional kinds of coverage helps in learning about the product that enhances the ability to perform better functional testing.
We are in Products and it is a enterprise level product. Client uses the software as per given directions.
Do you watch clients use the software? I am sure a lot of others would want to ask you the question I ask you. We don't have control over what the user does nor can we have. They do things that you don't know that they are doing and maybe the best, you can learn some of them.
But yes, if we start exploring life beyond functionality, then it will help us (testers) in thinking out of the box and increase lateral thinking skills, which indirectly helps in doing better functional testing.
Exploring beyond functionality WONT help in improving lateral thinking or out of the box thinking. Read and practice : Edward De Bono, Lateral Thinking & Think
@Pradeep - you don't know?? We have spy cameras. and we are spying clients. HAHA.. kidding.. dont mind.
ReplyDeleteYes, we are not watching clients. Also, they sometimes report bugs which are non-functional. however priority of these issues is very low.
Recently we carried out Defect leakage analysis, and found that we need to keep focus on Usability and Performance testing (Provided BA should get all these Performance related requirements from client).
But still, if you are in products, you need to keep focus on functionality.
@OUG,
ReplyDeleteYes, we are not watching clients. Also, they sometimes report bugs which are non-functional. however priority of these issues is very low.
How about being skeptical if the they are really low priority? Could someone be wrong about it?
But still, if you are in products, you need to keep focus on functionality.
Re-read the post after 10 days. Give yourself a change to live beyond functionality.
"Re-read the post after 10 days. Give yourself a change to live beyond functionality."
ReplyDeleteafter 10 days, your post will be read by 100s and it will get many comments.
Do you think, I will be convinced on that day?
.
.
.
.
.
Well Yes, your post and comments convinced today only. Now I am understanding your post and comparing with our leakage analysis with your post.
I cannot explain our leakage analysis here as it is confidential. But yes, apart from functional testing we need to keep focus on SOME non-functional aspects as well. (see my previous comment)
Thanks for your comments. These are helpful for me, as i am novice. Also, it will be a good idea if you post any case study here (if you want) through which you can explain - "even if the scope is functional, why non-function part needs to take case OR similar".
From my perspective, in the described scenario there seems to be a disconnection between the views of the testers and the views of the managers. As long as the testers do not start to /talk/ about their situation and their possibilities with management, this disconnection will stay, probably leading to dysfunction and de-motivation.
ReplyDeleteWeinberg discusses this in Quality Software Management Vol. 3 - Congruent Action. Here is a short summary:
All communication is based on the self position, the other position and the contextual position. When leaving out one of these three elements, you get incongruent communication. Leaving out the self leads to placating, leaving out the other leads to blaming, leaving out self and other leads to superreasonable behavior, leaving out the context leads to loving or hating behavior, and finally leaving out all three leads to irrelevant behavior.
In the described scenario above, there is a clear disconnection between what the testers think about themselves, and what management knows about the testers and what management thinks the testers think. So, the testers leave out the self position in their communication, leading to a placating behavior, and management does not know about the other position, leading probably to a blaming behavior.
So, in order to fix the communication problem there, the testers need to talk to their managers, explaining the impossibility to test completely. It's a hard and daunting task, since whenever I start to speak about testing, I can really see the ears getting shut, since no one is listening. Of course, this might be another problem, leading to the same symptoms.
@Markus,
ReplyDeleteYou are right. There is definitely a disconnect between testers and managers. I see this in many organizations and irrespective of ego caused it or not, ego definitely has been helping the gap to remain.
I should get to that book of Jerry soon.
Informative post,
ReplyDeletebut,had faced the problems in trying to implement some of these as Management does not even know what I was trying to explain and worst is they didnt care to understand..so,it put me in a bad position so had to go with the flow.Worst state of the Management in my company is that Test Manager/Leads do not even know what is functional ,non functional etc or system/inter regression and scopes of those testing...all they worried about is to make project to so-called GREEN status and keep dev teams(in fact,Test Management gets the things done as per Dev's instructions, so you can expect state /quality of the testing in this organization) and higher management happy. :-(
Very well written Pradeep.
ReplyDeleteOne question though, when and where can we draw line to make non-functional testing manageable. Non- functional test has a very large area to test and having limited time for every test project how can we make sure about the non- functional requirement as these requirements might be missiod by BAs or requirement management group due to their large scope.
Waiting to see the paradigm shift where there will be a balance between the functional, non- functional and alien testing :-)
@ Tarik Seth,
ReplyDeleteThanks for asking me that question.
Why do we test for functionality?
Our stakeholders have lots of questions about it and we are trying to answer them.
How do we draw a line of how much of non functional testing to be done?
Is by asking the stake holders questions about non functional aspects of the software and understanding what questions do they want to be answered. That would determine how much to do it.
Lets assume this:
Tester: Hey Stakholder, do you have an idea of how usability should be for this product?
SH: I have no idea. It should be good.
Tester: Well, what do you mean by good?
SH: I don't know to express it.
Tester: That's OK, do you think if you get some help to express it, you would be able to explain it to me?
SH: Oh yeah.
That's a point where we can offer some help to them provided we have educated ourselves on those areas.
SH: OK, so, maybe I want at least a few users to be brought in to be able to get a view of how they think is the usability of the product.
Tester: Hmmm! I am interested to answer your question however we don't seem to have budgeted for that.
SH: Right I see that point.
Tester: Well, we have been answering a question about functionality for a long time. How about pausing that just for a little while and focusing on these kind of tests. The good thing is when you are testing for usability, testers also have to work on these functions and hence would be able to report if something has gone wrong.
An automated check in place should help us too.
Great!!! thanks Pradeep.
ReplyDelete