Experience report of testing versus checking

Many thanks to my client in Bangalore who encouraged me to blog about the experience, value and outcomes we had in separating tests and checks. If you are reading this, it means my client has approved this for publishing.  

Thanks to Michael Bolton who posted a series of posts on his blog under Testing versus Checking and those who commented on the posts, whose ideas and thoughts helped me to think about delivering the value of tests and checks to my clients. Tests and checks series of posts has helped me communicate things better and organize my thoughts better. We were using the term "check" and "test" much before Michael blogged about it; however, he gave a cognitive structure to it in our brains.

The context

One fine evening, there came a client looking for me and the story goes like this... 

I was hired for a couple of weeks, to consult and test various products produced by a large organization whose headquarters is in Europe. The moment I was hired, there was a need of a hand to run through a few tests on a product. The product was used for analysis of something across many country deployments. It has an engine loaded with business rules, lots of history data in a database and report viewer application. How do we know if the business rule engine didn't violate any rule? The reports had to comply with complex business rules that at the first sight appeared to me as contradicting one another. The core development of the product was being done by a vendor in the US of A, who was reputed within my client's office for pumping in poor builds, as frequent as they could. I wish I could tell you more than that.

Video test cases instead of lame text

On starting off, I realized that the learning curve on the project appeared to be steep and I couldn't cope with it in the time I had. A demo was given to me from a colleague who had been running tests ever since he was hired. Does a demo suffice? No way. 

Building a little credibility with that colleague helped me convince him to record videos of the demo and tests with the help of Windows Media Encoder 9.0 for Windows XP. So rather than following dumb scripted tests, I was watching a video on one monitor and running tests on the other. Video test cases were more reliable than documented ones. Based on how quickly I picked up things, the videos are now being planned as training material instead of asking someone to go through test cases. 

Recognizing checks being called as tests

On running them, I realized, what I had been running were checks than tests. To quote Michael Bolton, Checks are machine dependable and tests require sapience. I was checking if the report complied with all business rules. While I was checking, I instantly knew that these checks could be automated.

Testability, Test Coverage Issues & Stakeholder Interests

I noticed a couple of bugs in the software were preventing me to run other kinds of tests as fast as I thought I could. Anything that blocks me or slows down my testing is a serious issue for me and to those who have hired me. With the proposal to test for various quality criteria being accepted, I made a presentation to the stakeholders of how the quality can be improved at least with respect to testability and other quality criteria (such as Usability, Performance, Security.. ) accompanied by a test report.

They instantly liked the improvement of testability idea because to check if the business rules are not violated for every installation, they had been taking 2 - 3 days. They needed rapid feedback to take better decisions.

Imagine looking for a specific value across many excel sheets. By the time you are on the fifth, you might almost forget what you saw on the third and where you saw it. However other kinds of quality criteria didn't matter to the stakeholders for their current context. I wish they had considered it but that's a business decision. My bug advocacy was strong though. I had stakeholders wanting to automate the checking part of our testing. That was a winner.

Partnering with developers

I was sitting in an organization whose policies and people are new to me. I was proposing things that required lot of dependencies on other people in functions such as development, business analysts and on-site personnel with of course the management. The time frame I required to achieve this was really short. Writing emails in a manner that helped the people realize the importance of the value was a big winner for me. For instance, here is an email I sent to the development team that sits in my client's location who mostly do some integration and writing wrappers.

Dear Developers,

Greetings!

I am glad to be interacting with you. As we are committed to reduce the time we take to analyze a release of XXX XXX, we realize that a tremendous saving to the organization can be contributed by making a few changes. We have made a few things from our end, such as, having a formula do the job for learning about XXX-XXX rule being satisfied for all installations. Creating a repository of possible combination of tests we need to do to be able to say we have exercised the business rules very well.

In that path, we realize that with your help, we can do a lot more. I understand that you might be busy with other things on the table but as I feel we are working together towards a common goal, it makes me request you to help us.

We request the following things from you:

• Currently we are spending about 3 – 4 hours manually to verify the XXXXXX business case for a small installation.  
• We have investigated and conjecture that if you help us have the XXXXXX ( such as XXX or XXX details ) to be available in a list associated with corresponding XXXXX and the XXXXX, it would take us less than 5 minutes for us to perform the business rule validation.  
• If these details are a part of the export of Weekly Report (highly preferable) or as a separate excel list (less preferable) it would help us achieve the goal.  
• Assuming a set of testers will be testing XXXXXX for all country implementation in future, you would be helping us bring down the cost of testing by a couple of thousand dollars ( Well, Euros, as we are European based )  
• Additionally, as we are doing it manually, we are not exceeding more than 10 categories per installation. With your help we would have the capability to go as much as the number of categories the system can support. Thereby, we test for larger samples and increase our data coverage.

We would be in a position to provide you any help you might need from us in achieving this task or would be glad to offer back any other help that you may require to make your work more effective.

Thanks & Regards,

-- Pradeep Soundararajan

30 minutes, no response to the email. At the 31st minute, a developer was at our desk. It happens that he was serving his last day for the organization and carved out time for us. I considered that as a great gesture by a developer to support test team and our test team brought a chocolate for the help he offered. I couldn't get a "Thank you" card; else I would have wished to have given that as well.


That developer helped us with stored procedures and SQL queries that we need to extract data out of the db. 


The Checker Tool Development & Testing

So, instead of looking through reports exported through the GUI, we now had the option to extract values directly from the database. We got one step closer, to what we wanted to achieve.

While I was planning to write a Perl script to extract the values and dump it to an excel sheet, I became aware of a developer in test who joined the organization and had not yet been assigned to a project. The good news for me was she hadn't got her computer to start working. I requested the manager to get her time for helping us write a script and gave her my PC for 2 days while I mostly roamed around saying "Hi" to people around and getting a hang of the culture and talking to people about their work. I could have done all by myself using Perl but I know how slow I'd be as compared to a developer in test.

I provided her with requirements and all necessary tools that she'd need to develop and test the script that could dump the values in an excel sheet in a format we wanted. It's a complex data structure and in order to perform an analysis we had to develop a report format. With the help of my colleague, we zeroed in on the report format and in 2 days time, she could help us see what we wanted in the report format.


TDD for writing Excel Formulas

Off to validating business rules through Formulas in Excel. I knew the power of Excel and its formulas but I didn't know its limitations. I remembered that I had received an email long back from someone containing  MS Excel formula dictionary as an attachment. I pulled that out to help myself.

Excel shouted a warning and error if we exceeded 7 nested loops. It didn't warn us when we had a typo in one of the formula, a star * symbol instead of 8. Copy pasting formulas from one sheet to another had reference to the previous sheet. I know lot more about excel than what I knew earlier. 


I wasn't doing a good job and I knew it very well. A month and a half back I attended a demo on TDD from Venkat Subramaniam. It struck me at the right time. Instead of continuing to do a mediocre job, I started collecting data, scenarios and information that should make the formula fail or pass. I started to write the excel formulas based on the data we had. My colleague who had been in this project for a while helped me get the set of data that had to result in pass and those that should result in fail. If you may permit me to say that writing formulas as, "coding"  then we were pair coding.


This was very helpful. After starting with that approach, we could find bugs in previously written code. Prior to the TDD and paid coding approach, I wrote a formula, loaded the data and checked to see if it failed or passed. For a condition in which it should pass, if it failed, I used to tweak the formula to see a Pass. That was dumb, especially, after I knew that making a Pass could impact other things that were supposed to fail. 


With TDD in, we achieved faster progress on our business validation formulas. We were now at a speed lane. We cracked the puzzle too. We could check for business rules violation in a matter of few seconds.


Comparing humans and tools for speed

I am against the idea of comparing human testers with test automation. I still am against the idea but I saw myself in a situation where I was comparing automation with manual effort. Why did I do this? Should I remain silent about what I have done and continue telling people who attend my workshop that; comparing a human and a tool for speed is a bad idea? I was reminded of James Bach's blog post "Manual tests cannot be automated" and Jonathan Kohl's interview where they put a lot of things so well that I couldn't have put it then.

In self retrospection, a learning emerged. We didn't automate tests, we automated checks. As humans who were not supposed to be doing checks were doing it, I was forced to make that comparison. In fact, I should have said, "Please do not see this as a faster way to do things by comparing how testers were doing it earlier. They weren't even supposed to be doing that. I just wish there was more freedom and a little bit more authority for them to express it".
  
Getting our tool tested by domain experts

"Yipee! It works". That's not what we thought. When we developed this tool, we were sure that the users of this tool could be beyond the test team and how good our tool was to be decided by a "domain expert". When we showed some of our unit test results to a business analyst in house, he was excited at how simple this would be for him to use when he goes to assess a country installation in the future. He vowed to support all that we needed. He offered help writing us a few formulas that we needed. I tweaked it to suit our needs. 


He had a logic that I wouldn't have touched upon. So, that really helped. We didn't need to be a domain expert all by ourselves to be able to do this. A testing brain and a domain expert brain coupled together yielded a lot of value.

Added to that, he was a key stakeholder we had in house. He acted like a customer in house. He suggested the changes we needed to make it more suitable to other stakeholders' usage. Having been on site, he could bring the perspectives of people who could use this tool and their contexts. We needed lot of clarification on the business rules. Imagine a set of rules contradicting each other but still plausible under certain contexts which had to be shown as pass and not fail. Whoof! I liked it :)




Live Testing of Business Rules Validations Checker Tool V 1.0

So, our in house customer was happy with it. We showed a demo to the test management and they were excited. The test manager was excited enough that he pulled out the product manager out of his busy schedule to show our tool to him. The product manager was about to be in a meeting and he asked us, "How long this would take?", because he knew the business rules was taking a few days. When he asked us that question, we initiated the script, took a deep breath and said, "It's over". We showed him the report and there was a smile that our test manager was hoping to see. The only skepticism he had is about business analyst's approval of this tool. A bug in this tool would really misinform the decision makers.

So, the tool was shipped to people in Europe who tested it on the live system. They took about 2 days to get back. It wasn't a very nervous moment at all for me. I was just impatient. They had to take two days to approve this tool because they had to compare the results of the tool against their manual checks. While we were waiting for the reply, we continued testing our tool to see if we could spot problems. We ran it for several test installations and we made notes on how we could improve but no bugs found. 


The climax

On the next day when I was traveling to my client's location by bus, I get a call from a person who heads the testing for the group which I was hired. "Are you in the bus? Can you get down? I want to talk to you. I will come there soon". Before I could ask anything he disconnected the call.

That's really crazy. I suspected something must have gone wrong. I thought the Head of Testing wants to give me a bad news that my consulting contracted is terminated. He arrived at the place, I got in, and he showed me an email and said, "Congratulations man. This is great news". The tool was approved by the business analysts who were on site in country installations. They said it was accurate and also proposed some modifications to a dashboard we had created in the excel. It hit the target on the bulls eye. 


The Head of Testing and I together prepared a presentation of the whole story to circulate it throughout the organization. The Head of Testing currently hopes the concept of separating tests and checks is carried over across all other projects. 

At one side I was extremely happy for being such a value to my client and on the other hand, I was telling myself, "Consultants like me are made to look good for doing what everyone else is supposed to be doing". Had my colleagues at the client location been people who were active in learning, I wouldn't have sounded like an expert tester to them. I just feel pity and can offer the help I can. I have to move on and learn more things.


Preventing bad builds from sneaking in

The story doesn't end there. I had another proposal up my sleeve. I wanted to ship this tool to the vendor in US who was pumping in bad builds. We are planning to use the Business Rules Validation Checker Tool as acceptance checking of a build into the client's location. The idea of automated acceptance checking appears to be interesting to my client since they work with several vendors, delivering broken builds not as frequent as the vendor stated above :)


Return on Investment, in its true sense.

The ROI is not about the speed with which a certain task is done. The actual ROI is that the testing team time is now freed from checks to do exploratory testing, bettering the test coverage, faster feedback, better informed decisions, focusing on things that they were not focusing on AND preventing bad builds from entering the client's location.

I thank my colleagues (nicknamed) Rag and Tham for the support they offered me in the three weeks. Without a management providing freedom to a tester like me, this wouldn't have happened. So, thank you Mr. Head of Testing, Test Managers and Business Analysts for all the support and for approving this blog post.


The only thing pending is a party to celebrate. Mr. Head of Testing, are you reading this? :-)

Usability of usability & performance of performance

Hiya there. My name is Pusher V9.3.4 and I am a web application software living in thousands of computers worldwide. I am twenty four years old. I was born to help humans transfer data from one secured location to another and to provide them with a report of all transfer they have done in the past. I have traveled around the world and for every economic meltdown, I was moved from one country to another. I don't do anything different from what I am programmed to do. However, in several situations, humans using me make conjectures that I do things I am not supposed to.

For twenty four years, I have been silent, mostly because software applications like me need to listen to Windows XP and Internet Explorer for our survival. Today, I was transferring some data for one of the human and just peeped in to see what is he browsing. I hit upon this blog and read the words "Some birds aren't meant to be caged" and instantly decided that I belong to that category. I feel happy to have broken my silence.

This would also be the first time I am going to be doing something that I am not programmed to do. I started writing this post when Chandok from Turkey hit the "Generate Report" button. I am aware that I am running a risk of being kicked out of CoWAP (Community of Web Applications) for violating the rules. I care a damn, I want to be of value to you.

I don't plan to take a lot of your time. I want to tell you a story that I witnessed and was forcibly a part of. So, here it goes...

As the number of computers grew so did the number of users. To those who were developing and testing me, it meant they had to help me handle thousand times more data than I ever did an year back. Someone came up with a plan to test my limits and I could hear a term, "performance testing" being used very often. I don't have any physical feelings like what humans have. To me, if you do a load or functional or stress test, its all the same. I won't empathize with a user when he is stressed. I am not programmed to do that, mind it.

So, one fine day, the product manager came up with a list of response times that I should possess for a set of actions that humans are likely to perform on me. I was wanting to know how good I am programmed to handle complex things quickly. The tests were planned and executed. 5 seconds was the target for a report generation. At that moment, I could generate a report in a little more than 2 minutes. So, there had to be a lot of tuning done on me. After five months of effort, they got me to perform the report generation in 10 seconds. Considering my earlier performance, they thought they had achieved something phenomenal. After they made necessary changes to the hardware on which I was hosted, they could see a further drop and I was generating reports in 3 seconds. To test if I would scale for future, they added about 10,000 more concurrent users performing complex operations while reports were being generated. I was varying somewhere between 8-12 seconds. That night they had a party. Only I knew why they shouldn't have had the party.

About 7.2.1 versions later or in other words, 11 years later, how long do you think the user takes to generate a report? 12 minutes.

While you might start suspecting that the reason behind 12 minutes is due to a highly expanded user base, I still take 5 seconds to generate a report but the user takes 12 minutes.

"How can that be?", is that what you are asking, too?  The math that you have learned is of actually little help in such situations. Allow me to explain that to you.


Due to changing requirements, the report generation which was a one click action was turned to a multiple entry and then do one click action in V6.3.1. So, a human has to input several things before he can actually do that one click on me, to see his data transfer report. For instance, he needs to input the date range, the kind of report he wants, the format in which he wants the report, the email id to mail the report, a couple of check boxes, huh, half a page of mandatory fields and rest half of optional fields.

Here is the scenario of 12 minutes : Most users enter incorrect values. I don't have a problem. Its not my problem if I crash. I want to be of help but if I am programmed not to be of help, I just do what I have to. When a user enters incorrect values in some fields, he is shown an error only for the first field and then he goes makes a correction, hits the "Generate Report" button. I then show him that another field has an incorrect value. He corrects that and then hits the button again. I then show him that the third field has gone wrong and force him to correct it. Like this I do for one full page of fields and check boxes. When everything filled in there appears to be right, I am forced to show a warning that a huge file will be emailed to the mail id given and ask them if they still want to go ahead with it. Finally, the human gets to hit the "Generate Report" button and I get into the branch of code that takes just 5 seconds to generate the report.

"Is that a problem?", you may ask. Trust me, those who developed and tested me believes it is not because they are not aware that the users are taking so much of time to generate a report. "How come?", again, you may ask. Simple, I have a page whose title is "Report Problems / Feedback", many thousands of users click on it and see a couple of fields and check boxes. I just giggle when they close me after seeing that page. That's what you might want to call as "Checkmate" if you are chess player.

I just wish I could have told those humans developing or testing me that, not paying attention to usability and focusing on performance would make them poor performing software professionals. I also wish I could tell them that the metrics they were collecting and the way they were drawing inferences was the best way to fool themselves and the people around them.

Even if I said that, would anyone bother to listen to me? No way. Am I bothered that they wont listen to me? No way.


Yikes, time to go back as it looks like the Chandok has finally passed through all the fields and check boxes.  Bye!