You have blood, it has a pressure and it has to be monitored if you want to keep yourself healthy and living long enough. Blood pressure is directly related to heart and its function. As you all maybe aware, most deaths in this world are due to heart attack.
We are so bloody lucky. If we were born in the 17th century and needed a blood pressure test, someone would have wanted to cut open an artery, insert a pipe and then measure how much rise and fall the blood has in that pipe. There is no guarantee we may survive that single test. Since 1855, the situation changed when Vierordt worked on a non invasive method of testing blood pressure. That invention has changed the world. One of the greatest physiologists of the nineteenth century, Johannes Muller said, "the discovery of blood pressure was more important than the discovery of blood". Thankfully, we don't need to be as great physiologist as as Muller was to say that the invention of non invasive way of blood pressure measurement has been critical to saving millions of lives every year.
Not just blood pressure. There were lot of medical tests that were invasive and inventions have helped us find non-invasive ways to test. If that had not happened, every time someone goes to a medical checkup, they would come back with a puncture in the body (if they were still alive)
The introduction of non invasive ways to measure blood pressure and many other measurements related to health are excellent examples of good testability. The MRI, CAT, Doppler, CT scanners are result of humans thinking of good test-ability mechanisms. This has helped doctors to do better and faster diagnosis of health problems and indirectly save lot of lives. Without good test-ability layers in health sector, our average life span would not be as high as it is today. Testability has improved so well that my grand mother uses a device at home every week to test her blood sugar levels, despite it being invasive. In the future, an iPhone or an iPad may be equipped with apps and hardware that may do non invasive tests and auto report to the doc or auto fix an appointment based on the reading.
Testability, is "the not as famous as it should have been, yet a fundamental thing to help in diagnosis". Be it in tests for health or tests for software.
My introduction and experience of testability
I first heard & read about testability 3 years after I became a software tester from James and Michael. Once I learned testability, I found that the root cause for many problems in testing software when investigated branches back to testability. No, this is no obsession or confirmation bias towards testability.
In 2008, I got a call from a company in Bangalore. Somebody had pressed the Panic button in the organization about performance of the product and they could only find me to be the independent tester available to give them a neutral opinion. I hadn't done much work on performance but I took it up. They wanted me from evening to next day morning because they had to make a release decision the next day. They gave me a feeling that I was Tom Cruise of Mission Impossible who has to accept a mission midst of nowhere. I took it up. On the way, I browsed through Scott Barber 's work and called Rahul Verma and asked him if he would be fine to assist me on phone if I were to call him midnight :)
Before I could start testing for performance or thinking about calling Rahul Verma, my opening move was - is the code testable for performance. It was not. Tool integration did seem like a project in itself. Doh! Panic, for sure. Also, it couldn't take 5 users, why load the poor guy with 10,000?
Several years after that, I met a bunch of "performance testers" of large services organization and I probed them on their experience of panic and call for performance testing. They had a similar observation as I had, although they didn't use the word "testability".
You may have read this experience report from me on checking and building check automation. James Bach blogged about it and made specific mention of how he perceived this story to be about testability.
Using versus controlling
About two weeks ago, we were testing a web application used by just a few hundred million people around the year. We found a bug in production which is of "Oh my Gosh" types. Fortunately, we had kicked in a video recorder and we captured it. We have the evidence that it exists but we are finding it difficult to reproduce it. No, not that we are not aware of what we did but this particular bug occurs after a specific error from the server. To disprove our own assumption, we need to control the server. In some large organizations there exists an IT infrastructure team that controls the staging environments and testers don't get access to control it. They just use it and are happy just /using/ it. Without being able to control system under test, merely using it won't help test better.
The sin of closing a bug because it can't be reproduced & its link to testablity
Going back to my days of being managed as a tester, in several organizations, my team was asked to close bugs that are not reproducible. I have heard and seen plenty of test and dev managers do that. When I picked up some wisdom in life, I realized how foolish the decision of closing not reproducible bugs was. Some testers defend the decision but are unable to reproduce it. They know they have a problem, unfortunately, they don't know they are facing a challenge of not having test-ability.
Control power to testers
There are products that do not have GUI at all. Not having a GUI shouldn't stop someone from testing the way they would do if it did have a GUI. Of course, we could write code that tests but it is difficult to alter a test when its running through automation. The human brain is wonderful at doing that. I change tests as I see some information and get curious if I would get a bunch of different set of bugs and do a different coverage.
Finding Nemo exercise to help teach testability
Have you tried it? There is an exercise I created to help testers practice reverse engineering or what I thought was reverse engineering. If you want to download the exe directly, here is the link. There was a hidden lesson about testability in it. I now have enough evidence that most testers who worked on it in my workshop never said they were annoyed by the constraints I had put in or asked me how they could get out of the constraint I had put in.
This, I believe is a small slice of how they work in organizations without asking for testability. I wish every tester was first taught testability even before teaching how to test.
Process Explorer as an example of value of testability
One of the utility I discovered through the help of my student Mohammad on Windows was Process Explorer. This gave me an amazing view of a list of printable strings in a running exe.
If you were to run calculator and then go to process explorer, you would see calc.exe running. When you right click and go to properties, you'd end up with multiple tab window open. One of them contains printable strings in calc.exe.
One of them said, "This operation may take a long time to complete". Wonderful. So, if this is the output I have to get, what should be my test was the question.
I did a bunch of tests to figure out that a factorial had the power to make calc.exe say, "This operation may take a very long time to complete". I then applied Ben Simo's FAILURE mnemonic and found how unhelpful the message was plus perusing through other printable strings, I could do a error message coverage for calc.exe. I found an impressive list of problems, rapidly. It was a wonderful feeling and I wanted it to continue across all projects I test.
I thought it is a luxury to have such visibility and testability tools in all projects for the coverage I wanted to achieve. I was wrong. I recognize it is a basic need. I know how to get it. It doesn't matter if I know, it matters if we all know.
There are ways in which I hope we could solve this problem and here are some of them
Testability stories
Believe it or not, there are two types of Agile teams in this world. The first one in which testers and developers have equal responsibilities to test and work as a team with high collaboration to resolve problems. I have heard and read stories that such teams exist somewhere in the world. Maybe they are in India, too. However, almost all teams I have consulted in the last 2 years that claimed to be that type of Agile, were not. Call me unlucky or call me an expert in consulting teams that pretend to be Agile, it would suit me.
I was wondering if we have a set of testability stories to be built while the product is being developed it would be of great value. As you know and agree, testers are also users of the products, there are no stories written considering testers as users of the product. It always focuses on end users (who are probably customers). Here are some examples of stories I am thinking:
We are so bloody lucky. If we were born in the 17th century and needed a blood pressure test, someone would have wanted to cut open an artery, insert a pipe and then measure how much rise and fall the blood has in that pipe. There is no guarantee we may survive that single test. Since 1855, the situation changed when Vierordt worked on a non invasive method of testing blood pressure. That invention has changed the world. One of the greatest physiologists of the nineteenth century, Johannes Muller said, "the discovery of blood pressure was more important than the discovery of blood". Thankfully, we don't need to be as great physiologist as as Muller was to say that the invention of non invasive way of blood pressure measurement has been critical to saving millions of lives every year.
Not just blood pressure. There were lot of medical tests that were invasive and inventions have helped us find non-invasive ways to test. If that had not happened, every time someone goes to a medical checkup, they would come back with a puncture in the body (if they were still alive)
The introduction of non invasive ways to measure blood pressure and many other measurements related to health are excellent examples of good testability. The MRI, CAT, Doppler, CT scanners are result of humans thinking of good test-ability mechanisms. This has helped doctors to do better and faster diagnosis of health problems and indirectly save lot of lives. Without good test-ability layers in health sector, our average life span would not be as high as it is today. Testability has improved so well that my grand mother uses a device at home every week to test her blood sugar levels, despite it being invasive. In the future, an iPhone or an iPad may be equipped with apps and hardware that may do non invasive tests and auto report to the doc or auto fix an appointment based on the reading.
Testability, is "the not as famous as it should have been, yet a fundamental thing to help in diagnosis". Be it in tests for health or tests for software.
My introduction and experience of testability
I first heard & read about testability 3 years after I became a software tester from James and Michael. Once I learned testability, I found that the root cause for many problems in testing software when investigated branches back to testability. No, this is no obsession or confirmation bias towards testability.
In 2008, I got a call from a company in Bangalore. Somebody had pressed the Panic button in the organization about performance of the product and they could only find me to be the independent tester available to give them a neutral opinion. I hadn't done much work on performance but I took it up. They wanted me from evening to next day morning because they had to make a release decision the next day. They gave me a feeling that I was Tom Cruise of Mission Impossible who has to accept a mission midst of nowhere. I took it up. On the way, I browsed through Scott Barber 's work and called Rahul Verma and asked him if he would be fine to assist me on phone if I were to call him midnight :)
Before I could start testing for performance or thinking about calling Rahul Verma, my opening move was - is the code testable for performance. It was not. Tool integration did seem like a project in itself. Doh! Panic, for sure. Also, it couldn't take 5 users, why load the poor guy with 10,000?
Several years after that, I met a bunch of "performance testers" of large services organization and I probed them on their experience of panic and call for performance testing. They had a similar observation as I had, although they didn't use the word "testability".
You may have read this experience report from me on checking and building check automation. James Bach blogged about it and made specific mention of how he perceived this story to be about testability.
Using versus controlling
About two weeks ago, we were testing a web application used by just a few hundred million people around the year. We found a bug in production which is of "Oh my Gosh" types. Fortunately, we had kicked in a video recorder and we captured it. We have the evidence that it exists but we are finding it difficult to reproduce it. No, not that we are not aware of what we did but this particular bug occurs after a specific error from the server. To disprove our own assumption, we need to control the server. In some large organizations there exists an IT infrastructure team that controls the staging environments and testers don't get access to control it. They just use it and are happy just /using/ it. Without being able to control system under test, merely using it won't help test better.
The sin of closing a bug because it can't be reproduced & its link to testablity
Going back to my days of being managed as a tester, in several organizations, my team was asked to close bugs that are not reproducible. I have heard and seen plenty of test and dev managers do that. When I picked up some wisdom in life, I realized how foolish the decision of closing not reproducible bugs was. Some testers defend the decision but are unable to reproduce it. They know they have a problem, unfortunately, they don't know they are facing a challenge of not having test-ability.
Control power to testers
There are products that do not have GUI at all. Not having a GUI shouldn't stop someone from testing the way they would do if it did have a GUI. Of course, we could write code that tests but it is difficult to alter a test when its running through automation. The human brain is wonderful at doing that. I change tests as I see some information and get curious if I would get a bunch of different set of bugs and do a different coverage.
Finding Nemo exercise to help teach testability
Have you tried it? There is an exercise I created to help testers practice reverse engineering or what I thought was reverse engineering. If you want to download the exe directly, here is the link. There was a hidden lesson about testability in it. I now have enough evidence that most testers who worked on it in my workshop never said they were annoyed by the constraints I had put in or asked me how they could get out of the constraint I had put in.
This, I believe is a small slice of how they work in organizations without asking for testability. I wish every tester was first taught testability even before teaching how to test.
Process Explorer as an example of value of testability
One of the utility I discovered through the help of my student Mohammad on Windows was Process Explorer. This gave me an amazing view of a list of printable strings in a running exe.
If you were to run calculator and then go to process explorer, you would see calc.exe running. When you right click and go to properties, you'd end up with multiple tab window open. One of them contains printable strings in calc.exe.
One of them said, "This operation may take a long time to complete". Wonderful. So, if this is the output I have to get, what should be my test was the question.
I did a bunch of tests to figure out that a factorial had the power to make calc.exe say, "This operation may take a very long time to complete". I then applied Ben Simo's FAILURE mnemonic and found how unhelpful the message was plus perusing through other printable strings, I could do a error message coverage for calc.exe. I found an impressive list of problems, rapidly. It was a wonderful feeling and I wanted it to continue across all projects I test.
I thought it is a luxury to have such visibility and testability tools in all projects for the coverage I wanted to achieve. I was wrong. I recognize it is a basic need. I know how to get it. It doesn't matter if I know, it matters if we all know.
There are ways in which I hope we could solve this problem and here are some of them
Testability stories
Believe it or not, there are two types of Agile teams in this world. The first one in which testers and developers have equal responsibilities to test and work as a team with high collaboration to resolve problems. I have heard and read stories that such teams exist somewhere in the world. Maybe they are in India, too. However, almost all teams I have consulted in the last 2 years that claimed to be that type of Agile, were not. Call me unlucky or call me an expert in consulting teams that pretend to be Agile, it would suit me.
I was wondering if we have a set of testability stories to be built while the product is being developed it would be of great value. As you know and agree, testers are also users of the products, there are no stories written considering testers as users of the product. It always focuses on end users (who are probably customers). Here are some examples of stories I am thinking:
- Testers would like to have this specific error occur in order to be able to perform tests related to recovery from the error
- Testers would like to be able to inject values that violates the front end validation because thats exactly what hackers would do
- Testers would like to integrate a code coverage tool to test what areas of code are they not hitting through the tests they think are great
- Testers would like to have test doubles to test for interaction of our software with third party components whose code is not in our control
- Testers would like to control the time delay in which a certain business process starts running to help achieve time based coverage
- Testers would like to try and add users to db via an excel loaded with values instead of having to key in a set of test users every time.
- Testers would like to have a log of every time someone clicks on the following links to monitor how people seem to be using it
- Testers would like to have statistics shared from the production environment on which pages get maximum hits and what are people searching for mostly and what times have peak traffic
- Testers would like to track all malformed requests that come to the server to monitor any hacking activity or trace of attempts to hack
It could be domain specific, product specific, testing generic and much more. Adam Goucher blogged about what Michael Bolton suggested as Design for Testability and Brian Marick has this wonderful essay on Testability and you would see that we all are thinking in the same direction - to help exploratory testers do better testing, faster and more frequently to help them do better test coverage.
If code needs instrumentation to accommodate testability stories and requests, it should be written in such a way that it does not disturb the sytem under test, can be removed or turned off before putting into production. Worried about this branch going to production - discipline of discipline can help prevent such problems or if you love automation, so be it.
The new idea here is about having identified the need for stories dedicated to "testers as users" of the product. This helps add layer and value to testing. When I say testers here and you belong to the real Agile team, please substitute it by "team" because it is everybody's responsibility to test, isn't it.
Review of this idea
I wrote a first level draft of this and sent to a bunch of people (from the huge set of people I could have sent to) whom I knew would help me out bad set of ideas out in public. It is their feedback that have helped me revise this to its present form and I would like to thank Paul Carvalho, Ben Simo, Rahul Verma, James Bach and Michael Bolton for their review. These people helped me identify the strength and the context in which this idea can be highly valuable. I am sure if they read this published piece, they would realize how different the first draft was compared to this one. That is how much influence their feedback has had on me and my writing on this one. I have thoroughly enjoyed the journey of thinking about this idea to writing down several drafts, researching and re-reading many articles and getting this to a point where you are done reading it :)
The new idea here is about having identified the need for stories dedicated to "testers as users" of the product. This helps add layer and value to testing. When I say testers here and you belong to the real Agile team, please substitute it by "team" because it is everybody's responsibility to test, isn't it.
Review of this idea
I wrote a first level draft of this and sent to a bunch of people (from the huge set of people I could have sent to) whom I knew would help me out bad set of ideas out in public. It is their feedback that have helped me revise this to its present form and I would like to thank Paul Carvalho, Ben Simo, Rahul Verma, James Bach and Michael Bolton for their review. These people helped me identify the strength and the context in which this idea can be highly valuable. I am sure if they read this published piece, they would realize how different the first draft was compared to this one. That is how much influence their feedback has had on me and my writing on this one. I have thoroughly enjoyed the journey of thinking about this idea to writing down several drafts, researching and re-reading many articles and getting this to a point where you are done reading it :)
6 comments:
It is indeed very different from the draft I saw. Very nice blog post and not at all a casual discussion on the important subject of testability.
A lot of great test ideas become impractical to execute because of lack of testability in the product. So, while we spend great amount of energy in thinking about good tests and testing them, if we as testers start highlighting what is stopping us from trying out certain ideas, it would go a long way in delivering better products.
Hello,
I am new to this testing & quality field and one line that really impressed me in this article is " I wish every tester was first taught testability even before teaching how to test. "
I hope every Quality/Testing Manager, Team Leader, Senior Executive & Trainer thinks in this way & the IT industry will get efficient & proficient testers.
Hello,
Was good to come across your blog..
When there are 'As a Developer' stories for infrastructure or architecture type issues in agile, 'As a tester' (as user) is a good food for thought instead of being lost in tasks or submerged as a part of function.. This definitely adds value to testability of the product.
Every layer of the product should have definite response avoiding non-deterministic behaviors.. saves time debugging the levels at which the system has given up..
@ Asha,
Thanks for recognizing and spending time here. I wish I could have interacted with you but there is no email id I see anywhere :)
The hindrance mentioned in the 'Using versus controlling' is the most common one. Seriously. Such big processes (mainly restrictions) that are laid in place stops us from doing things quickly. One who is good at excuses, would be happy not to control the server.
Need for a new blog post again.. Its been a long time :)
Post a Comment