BPO / Support / Call Center / Homemaker to Software Testing

Over the last few years, I have received at least about 40 emails and a couple of phone calls from people working in tech support, BPO, call center, homemakers asking for my advice to get a job in software testing or to make a career in it. I am tired responding to the same set of queries from different people. Now, that doesn't mean I wouldn't be interested to talk to people. I just hope they read this and have a different question or a situation they'd like me to address. 


However, this post is not just for those who want to change their career from BPO or Tech support to software testing but also to those who are hiring managers, interviewing testers and to all those who are aware that they are a part of the future of software testing.

1: Chasing dream versus chasing a day job

A friend of mine gave my number to his friend who works in a Technical Support job at a reputed organization and wants to move into software testing. So he called me to seek my help. I asked him, "Why software testing and not something else?" and he didn't have an answer. That is perfectly fine. He then said he wanted a day job and found software testing as an easy possibility.

I started to probe his dream of what he wanted to be before he landed up in Tech Support. His dream was to be something else. I then explained that moving to software testing might not help him feel any better than starting to chase the dream. He agreed and is now chasing his dream of photography.

2: Turning lemon to lemonade than faking it as orange

Some people have asked me if it would help to fake their experience of a tester of the product they were providing technical support just to get interview calls. I have helped them understand that there is a lot of value in presenting the truth than trying to show it as something else, get caught someday and be blacklisted by NASSCOM and a whole lot of companies spoiling future growth chances.

Having worked in several product organizations, I realize the importance of interacting and collaborating with support teams. If I were to hire a few testers for my team, I would definitely be interested to talk to a support team member. We did that in one of the product organizations I worked. I have talked to hiring managers of large and small product organizations who have done that. I think most of them are internal hiring and some rare cases of external hiring.


To all those who are considering to hire testers, stop doing what you have been doing all this while - hiring those who have been in testing only. Where were you before you started to do testing?


3: Learning software testing in 10 days OR Crash course about how to crash


With many folks wanting to learn software testing, lots of people are making money out of it. For all those in the world of software testing, do you know how many institutes are there per square inch of Ameerpet in Hyderabad who can teach testing in 5 hours if you'd like so and have enough cash? 


Not just Ameerpet, there are lots of chota Ameerpets that I have come across. These kind of training centers are a huge contribution factor for spoiling young minds in India. If God makes me rich, I shall wipe out each one of them.  These training centers run weekend batches for Tech Support folks and spoil their ability to learn testing. 


So, when folks who work in Tech Support and have attended such draining programs (yes, that was intentional) get in touch with me for seeking advice on job search, I have helped them to pick up two books: Testing Computer Software & Lessons Learned in Software Testing. For the most recent ones, I have also suggested Perfect Software & Other Illusions about Testing.


I also have suggested them to hook up with a tester every weekend and try some hands on testing or participate in open source projects for a while. For just one I have helped by doing a paired exploratory testing. One of my student in the Hands on Software Testing Training - India's first true hands on only testing training I delivered for Edista in 2008 & 2009 was in Support and he demonstrated his testing skills to his employer to be moved to testing.


4: Test Report instead of Resume / Profile


When my father started his first job after his Diploma in Electrical Engineering, he applied to the job with a CV / Resume. So, using a CV / Resume is that old an approach which hasn't changed much, except that he used a typewriter and we use MS Word and a Laserjet Printer. Lets try to change.


In the last 3 months, I have 4 emails from hiring managers in Bangalore, Chennai and a country outside India seeking help to hire skilled testers. One of the things I have suggested to them is to not ask people to send their resume unless it is accompanied with a test report. For hiring managers, it would be easy to see what kind of tester they want by looking at the test report. If the person says, "I am skilled at automating checks", so be it, demonstrate it and attach the scripts along with the resume. Needless to say without violating any Non Disclosure Agreement. Open Source software testing suits best. The interviews are actually discussion around the test report rather than "What is the difference between this and that?"


I am writing a whole big book on software testing interviews. A publisher just rejected the book but that's OK.


BTW, don't send me your resume and ask me to refer to those hiring managers.


5: Why some developer guys from India suck big time?


Nothing about their programming skills. I have done career counselling almost all through my career for others and myself. Hey, its a skill with which we are born, at least that's how we behave when someone approaches us for advice :)


So, having worked with some testing institutes, I volunteered to take up any work I could that would let me to speak with testers and potential testers for my own learning purpose. My blog, as you know, has brought me a lot of people with varied kinds of queries. So, I have some experience dealing with those developer guys who walk in months after their marriage, looking for a job for their wife.


These great developer guys come and ask, "I want my wife to take up a software testing job, do you offer job guarantee courses?". So, to the question, "Why software testing?", they'd without any bit of shame, answer, "I want her to earn and come home on time so that she takes care of office work and home work in a balanced way". 


One guy tried interviewing me to see if I know enough testing to teach his wife and help her learn testing. Not just me, one of my student, Arindam, was inspired to coach testers and works for an institute in Bangalore part time. That institute runs a special batch for home makers. Arindam shared his experience with me about the batch. He too said what I had already experienced. My advice to all housewife / homemakers whoever you want to call yourselves as is to read Parimala Shankaraiah's blog & Meeta's blog. Hey wait, there are quite a few others in India. Read their blogs to understand how passionate they are, how difficult it actually is to be good in testing and how they manage home and office work.


6: A break / sabbatical in career is just fine


Some women testers who take a break or sabbatical, try getting back to the industry but the industry treats them bad. Most hiring managers are blind in noticing people with break. They think they would be at loss of value if they hire them. 


I think stopping someone who is passionate in testing but had no other option than to take a sabbatical or break not getting a job is a big hindrance to the entire software testing industry. If I were Parimala Shankaraiah's employer and she needed a sabbatical, I would welcome her anytime she wants to come back. Not hiring her is like fooling myself and my company HR policies.


7. The actual meaning of "Our company is an equal opportunity employer"


There is a VERY BIG company who has an office even close to my home who has this statement, "We are an equal opportunity employer" in their website but didnt allow me to even apply to an opening just because I didn't have an ISEB/ISTQB certification. Let me tell this to you: I felt blessed by God to be not eligible to even apply to such companies because such company environments wouldn't have made my career strong.


The company which actually is an equal opportunity provider is one that selects its employees based on skills and not if they purchased a certificate. So, if you dont get jobs in such places, feel blessed, you really are. Keep demonstrating your skills and jobs will come to you. 


Santhosh Tuppad, my student with just one year of testing experience (and tons of experience in finding and reporting bugs) got an offer to be a Test Lead for one of the top companies in Asia. Although he couldn't take it up then but I just wonder if he had to be a Test Lead at some of those fake equal opportunity providers, how many white hair he should have had.


So, here are some points to ponder:

• If you choose to be misguided by what others say then you deserve it.
• If you have some other passion and want to be a software tester because you think its an easy job, you are spoiling an opportunity to help your children see you as their inspiration to pursue what they want to be.
• If you are OK to live others dream, don't question, just follow. Never crib / complain in life.
• If you want to test out testing, do so with open source projects and by collaborating with some good testers around you.
• No job is easy but all jobs can be done in an easy way.
• Fakers will get caught.
• Build your testing skills and demonstrate them.
• Attract employers don't always get attracted.
• Build your own brand. Get organizations proud about hiring you and not the other way.
• Many services companies count heads - not brains. Target tech start ups.
• If nothing works out and still you dream to be a tester, start your own testing services.
• Its OK to fail.
• Its important you succeed chasing your dream irrespective of whether your chase was successful or not.
• Chasing your dream is the only way you can know if you can really be successful

As and when I encounter more points or more different questions, I shall update this post.

Experience report of testing versus checking

Many thanks to my client in Bangalore who encouraged me to blog about the experience, value and outcomes we had in separating tests and checks. If you are reading this, it means my client has approved this for publishing.  

Thanks to Michael Bolton who posted a series of posts on his blog under Testing versus Checking and those who commented on the posts, whose ideas and thoughts helped me to think about delivering the value of tests and checks to my clients. Tests and checks series of posts has helped me communicate things better and organize my thoughts better. We were using the term "check" and "test" much before Michael blogged about it; however, he gave a cognitive structure to it in our brains.

The context

One fine evening, there came a client looking for me and the story goes like this... 

I was hired for a couple of weeks, to consult and test various products produced by a large organization whose headquarters is in Europe. The moment I was hired, there was a need of a hand to run through a few tests on a product. The product was used for analysis of something across many country deployments. It has an engine loaded with business rules, lots of history data in a database and report viewer application. How do we know if the business rule engine didn't violate any rule? The reports had to comply with complex business rules that at the first sight appeared to me as contradicting one another. The core development of the product was being done by a vendor in the US of A, who was reputed within my client's office for pumping in poor builds, as frequent as they could. I wish I could tell you more than that.

Video test cases instead of lame text

On starting off, I realized that the learning curve on the project appeared to be steep and I couldn't cope with it in the time I had. A demo was given to me from a colleague who had been running tests ever since he was hired. Does a demo suffice? No way. 

Building a little credibility with that colleague helped me convince him to record videos of the demo and tests with the help of Windows Media Encoder 9.0 for Windows XP. So rather than following dumb scripted tests, I was watching a video on one monitor and running tests on the other. Video test cases were more reliable than documented ones. Based on how quickly I picked up things, the videos are now being planned as training material instead of asking someone to go through test cases. 

Recognizing checks being called as tests

On running them, I realized, what I had been running were checks than tests. To quote Michael Bolton, Checks are machine dependable and tests require sapience. I was checking if the report complied with all business rules. While I was checking, I instantly knew that these checks could be automated.

Testability, Test Coverage Issues & Stakeholder Interests

I noticed a couple of bugs in the software were preventing me to run other kinds of tests as fast as I thought I could. Anything that blocks me or slows down my testing is a serious issue for me and to those who have hired me. With the proposal to test for various quality criteria being accepted, I made a presentation to the stakeholders of how the quality can be improved at least with respect to testability and other quality criteria (such as Usability, Performance, Security.. ) accompanied by a test report.

They instantly liked the improvement of testability idea because to check if the business rules are not violated for every installation, they had been taking 2 - 3 days. They needed rapid feedback to take better decisions.

Imagine looking for a specific value across many excel sheets. By the time you are on the fifth, you might almost forget what you saw on the third and where you saw it. However other kinds of quality criteria didn't matter to the stakeholders for their current context. I wish they had considered it but that's a business decision. My bug advocacy was strong though. I had stakeholders wanting to automate the checking part of our testing. That was a winner.

Partnering with developers

I was sitting in an organization whose policies and people are new to me. I was proposing things that required lot of dependencies on other people in functions such as development, business analysts and on-site personnel with of course the management. The time frame I required to achieve this was really short. Writing emails in a manner that helped the people realize the importance of the value was a big winner for me. For instance, here is an email I sent to the development team that sits in my client's location who mostly do some integration and writing wrappers.

Dear Developers,

Greetings!

I am glad to be interacting with you. As we are committed to reduce the time we take to analyze a release of XXX XXX, we realize that a tremendous saving to the organization can be contributed by making a few changes. We have made a few things from our end, such as, having a formula do the job for learning about XXX-XXX rule being satisfied for all installations. Creating a repository of possible combination of tests we need to do to be able to say we have exercised the business rules very well.

In that path, we realize that with your help, we can do a lot more. I understand that you might be busy with other things on the table but as I feel we are working together towards a common goal, it makes me request you to help us.

We request the following things from you:

• Currently we are spending about 3 – 4 hours manually to verify the XXXXXX business case for a small installation.  
• We have investigated and conjecture that if you help us have the XXXXXX ( such as XXX or XXX details ) to be available in a list associated with corresponding XXXXX and the XXXXX, it would take us less than 5 minutes for us to perform the business rule validation.  
• If these details are a part of the export of Weekly Report (highly preferable) or as a separate excel list (less preferable) it would help us achieve the goal.  
• Assuming a set of testers will be testing XXXXXX for all country implementation in future, you would be helping us bring down the cost of testing by a couple of thousand dollars ( Well, Euros, as we are European based )  
• Additionally, as we are doing it manually, we are not exceeding more than 10 categories per installation. With your help we would have the capability to go as much as the number of categories the system can support. Thereby, we test for larger samples and increase our data coverage.

We would be in a position to provide you any help you might need from us in achieving this task or would be glad to offer back any other help that you may require to make your work more effective.

Thanks & Regards,

-- Pradeep Soundararajan

30 minutes, no response to the email. At the 31st minute, a developer was at our desk. It happens that he was serving his last day for the organization and carved out time for us. I considered that as a great gesture by a developer to support test team and our test team brought a chocolate for the help he offered. I couldn't get a "Thank you" card; else I would have wished to have given that as well.


That developer helped us with stored procedures and SQL queries that we need to extract data out of the db. 


The Checker Tool Development & Testing

So, instead of looking through reports exported through the GUI, we now had the option to extract values directly from the database. We got one step closer, to what we wanted to achieve.

While I was planning to write a Perl script to extract the values and dump it to an excel sheet, I became aware of a developer in test who joined the organization and had not yet been assigned to a project. The good news for me was she hadn't got her computer to start working. I requested the manager to get her time for helping us write a script and gave her my PC for 2 days while I mostly roamed around saying "Hi" to people around and getting a hang of the culture and talking to people about their work. I could have done all by myself using Perl but I know how slow I'd be as compared to a developer in test.

I provided her with requirements and all necessary tools that she'd need to develop and test the script that could dump the values in an excel sheet in a format we wanted. It's a complex data structure and in order to perform an analysis we had to develop a report format. With the help of my colleague, we zeroed in on the report format and in 2 days time, she could help us see what we wanted in the report format.


TDD for writing Excel Formulas

Off to validating business rules through Formulas in Excel. I knew the power of Excel and its formulas but I didn't know its limitations. I remembered that I had received an email long back from someone containing  MS Excel formula dictionary as an attachment. I pulled that out to help myself.

Excel shouted a warning and error if we exceeded 7 nested loops. It didn't warn us when we had a typo in one of the formula, a star * symbol instead of 8. Copy pasting formulas from one sheet to another had reference to the previous sheet. I know lot more about excel than what I knew earlier. 


I wasn't doing a good job and I knew it very well. A month and a half back I attended a demo on TDD from Venkat Subramaniam. It struck me at the right time. Instead of continuing to do a mediocre job, I started collecting data, scenarios and information that should make the formula fail or pass. I started to write the excel formulas based on the data we had. My colleague who had been in this project for a while helped me get the set of data that had to result in pass and those that should result in fail. If you may permit me to say that writing formulas as, "coding"  then we were pair coding.


This was very helpful. After starting with that approach, we could find bugs in previously written code. Prior to the TDD and paid coding approach, I wrote a formula, loaded the data and checked to see if it failed or passed. For a condition in which it should pass, if it failed, I used to tweak the formula to see a Pass. That was dumb, especially, after I knew that making a Pass could impact other things that were supposed to fail. 


With TDD in, we achieved faster progress on our business validation formulas. We were now at a speed lane. We cracked the puzzle too. We could check for business rules violation in a matter of few seconds.


Comparing humans and tools for speed

I am against the idea of comparing human testers with test automation. I still am against the idea but I saw myself in a situation where I was comparing automation with manual effort. Why did I do this? Should I remain silent about what I have done and continue telling people who attend my workshop that; comparing a human and a tool for speed is a bad idea? I was reminded of James Bach's blog post "Manual tests cannot be automated" and Jonathan Kohl's interview where they put a lot of things so well that I couldn't have put it then.

In self retrospection, a learning emerged. We didn't automate tests, we automated checks. As humans who were not supposed to be doing checks were doing it, I was forced to make that comparison. In fact, I should have said, "Please do not see this as a faster way to do things by comparing how testers were doing it earlier. They weren't even supposed to be doing that. I just wish there was more freedom and a little bit more authority for them to express it".
  
Getting our tool tested by domain experts

"Yipee! It works". That's not what we thought. When we developed this tool, we were sure that the users of this tool could be beyond the test team and how good our tool was to be decided by a "domain expert". When we showed some of our unit test results to a business analyst in house, he was excited at how simple this would be for him to use when he goes to assess a country installation in the future. He vowed to support all that we needed. He offered help writing us a few formulas that we needed. I tweaked it to suit our needs. 


He had a logic that I wouldn't have touched upon. So, that really helped. We didn't need to be a domain expert all by ourselves to be able to do this. A testing brain and a domain expert brain coupled together yielded a lot of value.

Added to that, he was a key stakeholder we had in house. He acted like a customer in house. He suggested the changes we needed to make it more suitable to other stakeholders' usage. Having been on site, he could bring the perspectives of people who could use this tool and their contexts. We needed lot of clarification on the business rules. Imagine a set of rules contradicting each other but still plausible under certain contexts which had to be shown as pass and not fail. Whoof! I liked it :)




Live Testing of Business Rules Validations Checker Tool V 1.0

So, our in house customer was happy with it. We showed a demo to the test management and they were excited. The test manager was excited enough that he pulled out the product manager out of his busy schedule to show our tool to him. The product manager was about to be in a meeting and he asked us, "How long this would take?", because he knew the business rules was taking a few days. When he asked us that question, we initiated the script, took a deep breath and said, "It's over". We showed him the report and there was a smile that our test manager was hoping to see. The only skepticism he had is about business analyst's approval of this tool. A bug in this tool would really misinform the decision makers.

So, the tool was shipped to people in Europe who tested it on the live system. They took about 2 days to get back. It wasn't a very nervous moment at all for me. I was just impatient. They had to take two days to approve this tool because they had to compare the results of the tool against their manual checks. While we were waiting for the reply, we continued testing our tool to see if we could spot problems. We ran it for several test installations and we made notes on how we could improve but no bugs found. 


The climax

On the next day when I was traveling to my client's location by bus, I get a call from a person who heads the testing for the group which I was hired. "Are you in the bus? Can you get down? I want to talk to you. I will come there soon". Before I could ask anything he disconnected the call.

That's really crazy. I suspected something must have gone wrong. I thought the Head of Testing wants to give me a bad news that my consulting contracted is terminated. He arrived at the place, I got in, and he showed me an email and said, "Congratulations man. This is great news". The tool was approved by the business analysts who were on site in country installations. They said it was accurate and also proposed some modifications to a dashboard we had created in the excel. It hit the target on the bulls eye. 


The Head of Testing and I together prepared a presentation of the whole story to circulate it throughout the organization. The Head of Testing currently hopes the concept of separating tests and checks is carried over across all other projects. 

At one side I was extremely happy for being such a value to my client and on the other hand, I was telling myself, "Consultants like me are made to look good for doing what everyone else is supposed to be doing". Had my colleagues at the client location been people who were active in learning, I wouldn't have sounded like an expert tester to them. I just feel pity and can offer the help I can. I have to move on and learn more things.


Preventing bad builds from sneaking in

The story doesn't end there. I had another proposal up my sleeve. I wanted to ship this tool to the vendor in US who was pumping in bad builds. We are planning to use the Business Rules Validation Checker Tool as acceptance checking of a build into the client's location. The idea of automated acceptance checking appears to be interesting to my client since they work with several vendors, delivering broken builds not as frequent as the vendor stated above :)


Return on Investment, in its true sense.

The ROI is not about the speed with which a certain task is done. The actual ROI is that the testing team time is now freed from checks to do exploratory testing, bettering the test coverage, faster feedback, better informed decisions, focusing on things that they were not focusing on AND preventing bad builds from entering the client's location.

I thank my colleagues (nicknamed) Rag and Tham for the support they offered me in the three weeks. Without a management providing freedom to a tester like me, this wouldn't have happened. So, thank you Mr. Head of Testing, Test Managers and Business Analysts for all the support and for approving this blog post.


The only thing pending is a party to celebrate. Mr. Head of Testing, are you reading this? :-)

Finding Nemo Answers: Shmuel, William & Bangladesh Testers

So, I posted a challenge for software testers in December through this post : Why testers need to learn to write code that works. It was widely tweeted, circulated and even translated to Russian.

At the end of the post was an exe file with a challenge in it. I was practicing coding and wanted to do it in a way that helps me and others. I probably thought many Indian testers might pounce on it but that didn't happen. Even the testers I mentor didn't seem to take it serious. A part of mentoring is to not get disappointed when your mentees are not excited about a thing you are excited or you want them to be excited. 

However, there were some pleasant surprises. Thanks a lot to Alexei Barantsev, Shmuel Gershon, William Fisher & Sajjadul Hakkim and his team at Bangladesh.

I am going to be presenting a review of Shmuel Gershon's and William Fisher's work. What about Bangladeshi testers work, then? Well, its in encrypted form which I will link in the bottom.

At this point, I'd like to warn you that if you plan to attempt the challenge, you might not want to read further. However, if you don't want to attempt but read the answers and review, please go ahead. Think for a moment, maybe two and see if you really want to read the answer before attempting it. If you don't know what the puzzle is then the following wouldn't make sense to you, so just play with this and come back.

So, you decided to read their answers. No problem, you still made a wise choice :)

If you want to read Shmuel Gershon's experience report and answer first before reading my review on it, please do so. Here is the link to his blog.

The Black Box Tester Approach!

Shmuel, "So I downloaded Pradeep’s application, and got to work! Rather than just trying to solve the puzzle, I looked at it as if the mission was: “This is commercial 'roulette' style game. Stakeholders want to know it the game logic can be broken/learnt, which could mean a substantial loss of money when people start winning every time.”

That's a nice start. He didn't see it just as a puzzle but set a bigger mission to himself and made it more interesting. The way he set the mission didn't contradict a bit to the mission I had provided yet was different from the actual. That's a cool thing to do in such contexts. I would have been more curious to jump in to the puzzle and treat the mission as it is. So, that's a learning for me.

• 1. First, I ordered my environment to allow efficient and organized work:
  
  
  
  
  
  1. Increased the Screen Buffer Size of the Command line I was using to 500 (it turned out that I would do well on less, even half, than that (see point 2.b). But it didn’t hurt).
  2. Renamed the executable to be shorter and without number version: ren fnemo_1.7.6.exe fnemo.exe
  3. Changed the prompt to something shorter, cleaner, and that gave me context about this task against the other Cmd lines windows open: prompt $Cfnemo$F$S$G

Testing Hint: Organizing your testing environment before you start will likely help you during your tests. After you’re in the heat of fight, it will be harder to stop and get organized.

Shmuel provides a testing hint to all of us that is important but not practiced often. I should say that I would jumped into it without bothering to do any setup because

• I tried some basic executions of the program, to grasp the feeling of what it does and how hard it is to find Nemo. This also taught me what are the messages returned by the application when Nemo is found, and when Nemo isn’t found. They’re “Found Nemo this time!, Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger” and “Ah! bad luck, didnt find Nemo this time!” respectively.
1. I also learnt that Pradeep used Perl and Perl2Exe to do the app.
2. Additionally, the application clears the screen at every execution, so the big command line buffer is not so helpful.

While trying to developer test this program, I found that not clearing the command line left over was a little bit irritating to me to perform my next test and hence I added a clear screen in the code. You might be interested to note that I clear screen at start because when my code erred, I didn't want to see it. A clear screen gave me a fresh look at the output and I was happy. It is possible that other programmers do something similar which impacts the usability of the product.

So, programmers aren't wrong unless they don't want their code to be tested.

The next time I write code, I shall keep this in mind and would try giving the user an option if he'd like to clear the screen every time this program is launched. That might solve one problem, probably without creating others.

Sarmila commented on the Challenge page about disassembling the executable. Although by learning all the assembly one can learn about the rules that move the fishes around, it would be extremely difficult and overkill. But this made me think on how Perl2Exe works… Maybe it just wraps the Perl interpreter and the Perl script together? If so, what if the script is stored internally in clear? I tried to open the app in an Hex editor, but no, the Perl script isn’t in clear text there, it is obfuscated.

• I also tried the useful Strings tool by SysInternals. My intention was to see if, maybe, the array of fishes could be seen at the app code, in order to learn the initial order of fishes. It couldn’t.
• That ended my cheating session  , from here on I used only a black box functional approach.

So, Shmuel came to a conclusion that the code might be obfuscated. Well, its not. Watch out for William Fisher's approach! However, Shmuel did try to cheat. When the programmers say, "Don't test this part!", that doesn't mean you don't do any tests there. A little bit of cheating could reveal bugs that remain even after the feature is moved to a testing ready state. Maybe, you might learn something about the software or maybe learn how the programmer stages his or her work. That insight is helpful, too. So, did Shmuel cracked without cheating?

I don't know, why not read further?

By now, I knew that Nemo changed places between plays.

1. Nemo also change places between invalid parameters (which may or not be a bug).
2. Moreover, the other fishes change places too, even their placement related to Nemos’ placement changes.
3. So I tried to see if Nemo will return to the same place in any consistent way or number of times. Running that is easy, you just enter “3″, “3″, “3″… and count where Nemo was found and where he wasn’t.
• I discovered there is regularity (for “3″, its every 2, 3, 17, 5, 10, 5…), but the regularity was irregular enough, and also changed for different positions. This was likely a consequence of the real logic, rather than the logic itself. This correlates to the movement of fishes, but does not cause it.

Shmuel starts with simple tests and that's very important. I don't know why our brains try to move towards complicating things than simplifying it. It looks like "simple thinking" is a skill and a must for tester. Puzzles, especially those which we were unable to solve were the ones which we complicated it so much that we put a puzzle2 over the puzzle1 and tried solving puzzle2 and yet tried getting the answer for puzzle1 through that. Solve it directly!

Another detail that helped me in the game was knowing the developer and the purpose of the challenge. This puts a lot of content in the context. Pradeep is a rather playful  and would put some tricks into it.

1. First I tried to see if Nemo could sometimes be in any placement bigger than 7 (there are 7 fishes, places 1 to 7, but there were no rules as to where the fishes were limited to be).
2. I tried in a toss of up to 220 attempts, and Nemo wasn’t in place 8 even once. So I let this idea on StandBy for now.
3. Then I tried… what if the “Minimum attempts” input at the beginning of the app affected where Nemo appears? 
   
   
   
   
   
   • 10 is the default for the “Minimum Attempts“. Nemo is at place 1 at the beginning there.
   • 11 attempts chosen: Nemo is at place 1 at the beginning too. Maybe it does not affect?
   • 12 attempts chosen: Nemo is at place 4! Bingo!
   • 13, 14, 15, 16 showed that the original places were cycling at ‘3′ intervals. That means that when trying to find Nemo on the default 10 attempts, it will be in the same placements as 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52…

Having understood some bit of me, Shmuel made a conjecture that I could be playful and do some tricks to fool you all. That's a kind of thing like trying to understand the psychology of the programmer and model tests around it. 

You also notice the OFAT heuristic? You don't know OFAT, no problem, its One Factor At a Time. Now, I don't need to explain MFAT, right? Shmuel, tried using the OFAT heuristic (although he might call it in a different name) to narrow down or learn about the behavior. It did work well for him. He found something valuable that looks like helped him proceed on solving the puzzle.

You’re seeing that “220 attempts” above, and thinking if I really entered each number manually.

1. By this time, I decided that I was in need of some automation to insert inputs to the application.
2. The original app had no apparent automation capabilities, so I decided to use the internal input redirection of the Command line: 
   
   
   
   
   
   • By using the “<” operator, I could redirect an input file “input.txt” to the app using “fnemo.exe <  input.txt“
   • I extended that to “fnemo.exe < input.txt > output.txt“, which gives us with a complete dump of results that can be searched with Notepad.
3. We had an easy automation now, and this can help our sapient testing. I was exploring, and using automation to help me try different things help me explore my possibilities and get a bigger picture of the situation when needed.

Look at that powerful stuff. A tester shouldn't be classified as a manual only tester or an automation only tester but the industry designations suck. Shmuel provided evidence that it is the context in which either of the approach is used. Well, manual isn't a good work, so I agree with Sapient tester, as James Bach coined. Testing is related to human brain not the hands that type the input. All good testers I come across, don't separate manual and automation testing. They see it as approaches that compliment each other.

However, lets see what Shmuel did after that?

Do you still want to continue reading? 

---

• With the knowledge and tool I had so far, I did a lot of trials in order to find where Nemo is at each iteration. This could give me a hint if he moved in any regular or consistent manner.
• So I made a big file of winning moves (a lot of trial and error went in it).
• It was now time to model the results:
• First attempts at modeling the results:
• I got this: Nemo’s position: 1, 3, 5, 3, 2, 7, 3, 4, 7, 4, 4, 1, 5, 5, 2, 5, 6… 
  
  
  
  
  
  • Nemo was certainly jumping around without any clear regularity, even when I did it twice the times there was no repetition

On the Second modeling attempt, I tried to see if a more visual model could help:

1	2	3	4	5	6	7
X
					X
				X
		X
					X

Even in a table with many more results, it made no sense.

"No answer is also a useful answer". The first time I said that was in one of Elizabeth Hendrickson's exercise that I volunteered to guinea pig. Shmuel did fine. He looked for a pattern in the output but didn't get it and that was a clue for him that there appears to be no pattern. I wish he had done more tests on it but then sometimes critical thinking is a trap.

I didn't want someone to find the pattern of where Nemo is, that easily. I set the trap.

I changed my approach. Looking only at Nemo was not giving enough insight. Plus, I knew the other fishes moved along, so there might be a hint in their placement.

• I got this table:
• 
  
  
  
  
  

  Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
  Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow
  SharkTooth	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles
  Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow
  Stinger	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet
  Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth	Nemo
  Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth	Flow
  Flow	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles
  Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger	Nemo
  Flow	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles
  Flow	Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth
  Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
  SharkTooth	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles

• That didn’t look good. Nemo moved spuriously, and the other fishes either jumped aimlessly or stayed at their place. Nothing made much sense.

Shmuel, while he reads might be surprised to know that he didn't have to do this table. The code does the table, I had enabled logging. Shmuel didn't probably pay attention to his C:\ folder :)

Must admit that version 1.4 didn't have the logging but Michael Bolton, yes, the singer and tester, did a Rapid Test and suggested this feature. Maybe Shmueld did extract this from the log file but his report doesn't suggest he did.

I still thought that looking at the other fishes will bring the breakthrough needed.

1. In order to learn the movement of the fishes, I did a table of their placement in relation to Nemo at every run. 
   
   
   
   
   
   1. i. This is what I got:
   2. 
      
      
      
      
      

      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
      Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
      Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
      Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger

   • See anything interesting? The rows repeat themselves alternatingly! So the fishes were not really moving around, they were following Nemo!
   • My take is that Pradeep has two different arrays: One for Odd rows, the other for Even rows.
   • This may not be true, but it doesn’t matter for us. When testing, we not always know what exactly the programmer wrote in the code, but we infer a mental model. If it suits the needs, it is a good model even when not the real thing.
   • Many times I think these fake mental models are even better than the real thing. It’s the best way for them to act as an intuitive Oracle when analyzing an application.

"Boy, you got closer". That was my expression when I read the above. Good going pal!

So, if you have got curious to see what he did after that, you must visit his blog. I am not going to be revealing it further but to spoil your curiosity, he did crack it.

Kudos Shmuel! I hope other testers congratulate this effort of yours. It was very inspiring. I wish I could say "You could have done this and that" but it doesn't make sense to say that especially after your brilliant effort plus you did crack it! Your excel sheet, shows the way!

Now to William Fisher!

Whitebox Testing Approach

Aren't you curious to see the whitebox approach in action?

William Fisher...

K





Key:









 ! = Bug
* = Action taken
? = Question to investigate
@ = Assumption
Thoughts before executing:
? Size of Array
Unsure if the size of the array
@ At minimum, it must contain 7 elements.
? "All fish change positions"
@ Do all fish REALLY change position?
Wonder if their position oscillate between a set number of actual configurations
Are the ordinal positions simply incremeted? decremented?
Randomization?
Seeded?
? Are their intial positions consistent between session?
! I understand the rules, but how do I USE this?
? Assuming that I'm to enter a number representing an ordinal position
zero-based?
 
 
Notice, he has a different style of reporting than Shmuel, yet it is equally very useful and understandable. Finally, what matters is; is the information provided in a useful and understandable manner to the target audience? If you meet that through any kind of reporting style, good job done. William appears to set the context right to read his report further and when I ask questions to myself if I do that, I don't. However, just because I don't do it and I see other tester doing it is not a plausible reason that I should do it. Knowing different ways is of help when I'd have to switch style from one kind of audience to another.
 
If I have held you till this moment, It is a big achievement. Not just mine but of Shmuel & William, too.Test 1: 
 
 
Track the fish at ordinal position 1 across successive attempts
> Minimum attempts is set to 10 but you may increase it: 0
> 10 attempts left
! I entered a number and received the message "10 attempts left". Was I only supposed to press enter?
---
> Guess the position of Nemo :0
> You probably didn't give any any input or your input was invalid;please provide a valid input
> Guess the position of Nemo :"
! Did that count as an attempt? How many attempts do I have left?
@ If the language uses zero-based, potential exists for off-by-one error if input is not decremented properly
---
 
 
I told you. I did tell you, its hard for a tester to avoid finding a bug :). You also notice he is asking questions that are different from the ones Shmuel asked. He might also have suspected if making an invalid entry or just hitting the enter might cause a movement of Nemo but did it happen? 
 
> Guess the position of Nemo :1
> Ah! bad luck, didnt find Nemo this time!
>
> 9 attempt(s) left"
! There is an extra carriage return (prior to '9 attempt(s) left'. Should that be there?
! Attempts are decrementing from 10. Opening screen says I can increase it. How do I increase the number of attempts?
@ Different code used to display max number of attempts, than what is used to display the remaining number of attempts
@ The use of '(s)' in 'attempt(s)' indicates that the code does not change output based on remaining attempts.
@ Refactoring possiblity: use one function to display the remaining number of attempts. This will accomplish the same thing
---
 
  
So, as and when tests are performed, William is thinking about constructing the code in mind. It is definitely a useful approach, at least to me. I always try to imagine how the code behind this GUI look whenever I am not exposed to it. Modeling the code, no matter how wrong, is a useful approach. I get test ideas different than what I had by modeling the code behind the GUI. Now, if there ain't a fancy name for it, don't worry. It doesn't matter.
 
 


> Guess the position of Nemo :1
> Ah! bad luck, didnt find Nemo this time!
>
> 8 attempt(s) left
End session:
Found Nemo with 1 attempt left
Having found Nemo, I've been given the opportunity to guess one more time.
It closes after 5 seconds; cannot mark the entire session.
? Does this program generate any artifacts?
? When an attempt is not accepted, do the array positions change anyway?
================
Test 2:
Manual run the same inputs in s new session.
Run session with FILEMON attached
End session:
No artifacts generated by the executable
Did not find Nemo in this session
? Is Nemo's position time-based?
================

 
 
This is cool. Suspecting a specific thing to be happening and basing tests on it and not being biased when seeing the output. Appears like a critical thinking approach to me.
 
 

Test 3:
Manual run the same inputs in s new session.
Run session with TSEARCH attached
In TSEARCH, HexDump shows some secrets:
- print "\nHave you reverse engineered the logic? If you haven't go on!\n";
. else. {print "Ah! bad luck, didnt find Nemo this time!\n";.
End session:
Need to dump ASCII Strings from exe file.
 
You expect me to talk about tools now? Right. Tools are to help testers. This is another example of sapient testing, I would say. Until we use a tool, we don't know if its going to be of much help. If you observe what William was doing, he was constantly worried about code and was also concerned about which tool could be of help to him. Tried Filemon and then TSEARCH. I didn't know about TSEARCH till William sent me the report. Wow! You got to do something to get a tester to talk to you. That's all education happens as a side effect.
Every software offers us enough clues to help solve the problem. It ain't an intentional design, I guess. I still like to think that way although I might be wrong. I am not going to say this to my client while testing for them or we wouldn't have time to talk about these.
 
And you'd want to know what happened after William got that clue? Watch out for the video.
 

If you were by any chance curious while reading this, I too am, for a different reason. I want to know what would happen when William Fisher discovers the things that Shmuel did and vice versa. I bet they'd appreciate each others effort and approach and would learn from each other. 
 
Shmuel had almost concluded that there appears to be no way to look into the array by opening the exe in the HEX editor but William's experiments violated that.

 
Both of them almost missed the log file. William discovered the existence of the logfile by looking to the code. The next time I should just pass on the .pl file instead of EXE, I guess. All beans spilled, ah.

 
 
What's the deal with Bangladeshi Testers?

 
 
Ever since I discovered SQABD.com , I am wanting to go there. I want to go talk to those testers there. Not got a chance yet. I am hoping it doesn't remain a dream. Sajjadul is a tester with thoughts that are widely respected within and outside of Bangladesh. He and his team took time out and discussed about solving this puzzle, after experimenting on it for a while. 



Watch the video: (Some part in Bangladeshi and rest in English)

Want to get your hands on Finding Nemo 2? Watch out for it in March.