Who is making software testers, dumb and bad?

Not so long ago, I thought there existed a set of testers called, "bad testers". I hated them. I wanted to punch them on their face and get their face to bleed. I wanted to become a powerful politician and kill them all and escape without being charged for genocide. I wanted to become a superhero and get people to fire them from their jobs. I wanted them to beg for jobs, money and survival. I thought that is the way to get them to open their minds for learning. All this should have shot my blood pressure up while those bad testers remained cool. They were untouched by my criticism and continued to think that I was an asshole.


Whenever I found them, I insulted them as much as I could till I realized that they needed more care from me than those whom I was already caring about. I started caring for them. My world changed and so did theirs.


I was always wondering how these bad testers are happy. Needless to say I thought I am a great tester and still continue to think that way. By "great", I mean, "just what is required". Today, you can be a great tester by being just what is required. Tomorrow, the case might change.


I tried shifting the question from "Why are bad testers happy about themselves?" to "Who is making these bad testers happy?" and "Who is preventing the bad testers to learn that they are doing bad testing?"

That's when I said to myself, "There are no bad testers. There are some who are forced to practice bad testing. The force is either internal or external or a combination of them".

That was an important shift in the strategy that helped me in my exploration of identifying what factors cause a tester to appear bad or practice bad testing.


Internal forces 
I mean, ones that the testers themselves are responsible for or have control over.

• Money more important than anything else: For some testers who are sole breadwinners of the family, they might internalize the idea that what works for others is a safer route to traverse than exploring new paths and risking their cash flow. They spend their life traveling those peoples route who themselves have followed someone else's route. All finding it to be safe and hence not wanting to change.

• Fear of losing the job: For some testers, losing a job means unbearable social pressure. These testers don't ever try to speak against anything to protect their jobs. Their whole life is spent on running just one test case - Is this the right time to shut my mouth? - to which the result always remains - Pass.

• Shallow ambitions in life: For some testers, their ambition is to never do something fascinating but just run the rat race, build a house, buy a car, get married & have kids. They also try to ensure that their kids continue to run the rat race. I am not speaking against taking care of the family but taking care of the family should be balanced with building high ambitions in life and working towards it.

• Victim of Rutherford-Bohr's experiment: Some testers, no matter what exciting stuff they are presented with, try to return to their most stable state of ignoring all the exciting stuff because their life is already happy (grounded). 

• Living someone else's dream: Some testers, don't have dreams of their own. They just pretend to have their own while they are living other's dream. Some live the dream of their parents and rest their manager's. Living others dream makes their life boring and they give up on almost everything, forget testing.

• Taste of early success causing a drift from continuing to learn - Mostly a very dangerous one. These kind of testers think they are on the right path and there is no reason for them to change. 

• Having learned that good testing is hard - Some testers acknowledge what good testing is but they also learn it is very hard to test well. Out of that, some of them make up their mind saying they are not in for such hard work because they think life is bigger than doing good testing. Nothing wrong but they don't seem to be doing to the big part well, either.

External forces
I mean, the ones who are responsible or has a power or influence to get good testing done.

• Head or Tails of testing - I have talked to at least slightly less than a hundred Heads of Testing of big, medium and small scale organizations. They have so much power to change things and yet they don't seem to be doing anything about it. I must also admit that some people are doing very well while most don't appear to be. Why don't these people take a break from their work, sit along testers on one of the project and test for just a couple of days to realize how hard it is and what can they do to help these testers do a great job. 

• The interviewers - At least people in India, when they are out of college, want to just learn enough to crack an interview. When interviewers emphasize on demonstration of memorization than skills, its easy for a billion plus population to crack them. Fakers get in, Genuine people might not.

• Testing institutes - Business demands scale, I agree. Scaling at the cost of quality of education is in my opinion, spoiling your own country's chances. Please read the book Outliers by Malcom Gladwell and more specifically Chapter Five - The Three Lessons from Joe Flom. You'd know what your business needs for future if it has to remain scalable.

• The experts - If you have great ideas, please price them a little lower for the first few years or based on the geography. You won't be considered cheap, trust me. Don't make money an entry barrier to someone who wants to get excellent at testing. 

• Commercial conferences - If you have have had good deals of sponsorship and paid delegates for a specific year, consider giving 80% discount to 10 people who cant afford it but want to attend it.

Combination of internal & external forces

When the external forces & internal forces combine, its a killer combo for bad testing

• Lack of speed in firing poor performers - If the people responsible to get good testing done are delaying in firing poor performers then the hope in the poor performer rises that he or she is doing well and should continue doing that. In at least half the organizations I consult, I get the opportunity to consult because they haven't fired the poor performers for a long time and something went kaput.

• Not paying good testers well - I have been to a few conferences in India where Head of IT or Head of Dev or Head of Testing are keynote speakers. Their speech is usually, "We have come to realize that testing is of great importance" but then they don't match the pay of the good testers they have to their claims. People call that "keynote". Can you walk the talk?

• Waiting till the year end to spend on training budget - Wondering why many organizations keep their training budget till the year end and not organize a training when the team needs it sometime mid year? As a side note, I wish, in India, the Learning & Development department, which is a separate entity in the organization is eliminated and every department becomes Learning & Development apart from what they do. 

• The book writers - When you write books that are not different from any other books that are available, you are re-iterating the point that the industry isn't changing. Many testers who accidentally pick up a book and skim through it read stuff that they have read a couple of years ago feel they are on track (and also end up not buying your book). Is that a message your book wanted to communicate?

I am 30 now. I am more curious about my age of 50 and waiting to get there, because I hope, I would have seen many changes - lots of positive ones. Mostly because the generation to which I belong or the generations junior to that of mine would have solved the problems I have listed and might have gone beyond that. I am not discarding the fact that the older generations have not solved it. There are dozens of them out of a population of millions.

When I tried punching just one bad tester I met, blood oozed out. Not on the face but in my hands for it was a mirror that I saw. 

The domain particle

The inspiration


The title is inspired from Angels & Demons where they talk about the "God particle". The topic is  inspired by so many testers asking or answering to, "How much important is domain knowledge?", "Can we test without domain knowledge?", "What happens if we test a product or technology without domain knowledge?" ... 


Every person who writes in public and in forums have been asked these questions about the importance of domain knowledge to test a product. Going through those forums, you'd know that there are answers like : Yes, domain knowledge is very important to test a product OR Oh yeah, you know without domain knowledge I couldn't have found the show stopper I found a few days back...and others saying, "I did manage to learn quickly, so it doesn't matter as long as you too can", "I think it depends on what you are testing"... 


Passing time by failing to respect others time


The question I have to most testers out there is. If we, who reply to all those questions of yours, say that it is very important that you should be an expert of the domain to test the product, are you really going to become one or even try to? 


You are aware that bug investigation skills are important, without we having to say that, what have you done about that? or would you go pick up the test framing skills?


I am convinced that there are many hundreds and thousands of testers who are a trap not just to themselves but to others wanting to help testers. Every testing related forum is infected with people asking questions who actually don't intend to do anything about it despite unintentionally (or maybe intentionally) wasting a lot of passionate testers time & energy. There are genuine people in there but in a rare occurrence. Most genuine people aren't asking questions but answering them and not all who answer questions are genuine. You may want to consider me among those who are answering questions but not genuine, if that pleases your ego.


I have respect for CDT mailing list and Software Testing Club. There are serious people, asking, answering and watching. I just hope there are a few more forums like that.


Why do we have labels?


Some testers are in a trap of calling themselves "Telecom testers", "BFSI testers", "Web App testers", "Mobile App Testers", and what not? The question I have is, should it matter? BTW, I call myself an exploratory tester because its an approach I follow to test any software & not a domain.


As how programming is a mindset and a good programmer wouldn't call themselves "Java only programmer", testing is a mindset too. Oh, there is a skillset along with the mindset.


Henrik Andersson of Sweden is going to do a Webinar on October 12 : Do we need labels? Are we all not testers? I read the abstract of the webinar and was excited. I envy Henrik a lot. He is the guy who is blessed to be in the happening places - be it PSL, AYE or CAST. Other than that he is the energetic test consultant & partner at a testing consulting firm based in Sweden. I think you all should register and listen to what Henrik has to say. 


Henrik made me ask these questions: Why am I calling myself an exploratory tester, rapid tester, and whatever fancy stuff I had been calling myself when I know all testing is to some degree exploratory? What is the need for me to do so?


I need a label to differentiate myself or to communicate to people what I specialize at. My label helps my potential clients to gain interest to check about my services based on their needs. My label acts as a filter. People know what questions to shoot me. So far I have received very few emails from people asking if I can help them record and play QTP scripts. Most emails are about seeking help on how to develop themselves, think & test. I love to talk to them because they teach me things I want to learn.


So, I see label as a filter but I am against the labels that filter my own opportunities to learn. For instance, if I call myself a "Multimedia Tester", I would have blocked my learning on many technologies and contexts that I have experienced. I am starting to use "Brainual tester" these days. I am also giving a lightning / lightening talk in GTAC this year about it. James calls it Sapient and I call it Brainual. We both worship the same God but we call them in different names. I prefer the word "Brainual" because it is quicker to replace the word "Manual" to those who see "Manual" as a hand activity of testers than brain. People are used to saying, "Lets automate those manual test cases" but I expect the reluctance to set it to say, "Lets automate the brainual work thats being done now".  Their own ego would hurt them. They don't want to be seen as a fool.




Filter-less


Being a consultant and added to that the twist and turns I have had in my testing life has got me to test software ranging from wireless, mobile applications, medical devices, multimedia, Retail, CRM, desktop applications, dating applications, billing solutions, video surveillance, stock market, auction systems, kiosks, cloud computing, testing tools, games and what not. I have never bothered what domain each of them belong to as long as I have 

• An understanding of general principles of how software works (which I constantly refine)
• A skill to quickly learn and convert the learning to tests & churn more learning out of it.
• Ability to build models of learning to speed up my learning.

I try understanding how people who invented things might have thought. I fancy thinking that all these technologies have emerged out of observing something. Some of the key observations about human being and living things have led to many inventions and discoveries. Should I tell you that birds were an inspiration for Wright Brothers?


The human connection to technology


# 1 DHCP as to how it has evolved out of human behavior : When we go to the theater to watch a movie, we ask the personnel at the ticket counter to check if there are tickets available for a specific movie (sending request from a client to the server) of what we want to watch and our preferred time (details about client to see if server has anything for us). 

The ticket counter personnel in the theater gives us a ticket based on availability (assigning an IP address from the available pool) with the seat numbers to watch the movie.

In the ticket, there exists a seat number ( IP address ), duration of validity ( lease period ). If you prefer to watch the movie again and in the same seats ( Static IP ) you need to advance book or renew your tickets  
( ipconfig / renew )


# 2 Why do connectors have a male pin & a female pin? How did gender come in to technology without having observed humans and how they interact?


I have a huge list of things that computers/ technologies do which are based on how humans communicate or how humans could have. Think of SIP protocol or a client server host architecture and relate it to human communication. You'd probably enjoy as much as I do.


Simplification


In Rapid Software Testing class of  Michael Bolton & James Bach, there is an interesting dice exercise. I didn't crack it the first time but on learning the meta pattern, I am likely to crack it for any pattern or will get closer to solving it.


I have my own version of the dice game but with a different learning objective - simplicity. Based on running the exercise on thousands of testers in India (and a few folks outside India), I am making a conjecture that I see people struggling to cope up with simplicity. 


The first set of thoughts that comes to a human mind is not necessarily simple and the next set of thoughts are usually more complex than the previous ones. So, I think humans keep building on the already existing complexity of their previous thought.


Unless you train your brain to think simple, you are unlikely to crack many things you can.


The reason I am telling this here is because those who set out to learn a new domain forget that there are simpler sub systems of the domain they already know either because they also exist in other domains they have tested or have used such products extensively. You may want to figure out the meta pattern of software & how they are supposed to work. 




Practice to be fit

Quoting Parimala, "What you know is not as important as what you can do with what you know". 

I run an exercise in my workshop in which I allow people to do freestyle exploratory testing. At the end I ask them the techniques they consciously used while testing. Although many of them know many techniques to test, at least the first 10 minutes of most testers freestyle exploratory testing is, "Clicking here and there to see if a bug dances out on the screen" If a few bugs do dance, "Wow, you see I did exploratory testing". I think of that as an inferior self standard setting to understanding exploratory testing.  


How do we use what we already know to achieve better results? PRACTICE!


The biggest shame for ISTQB / CSTE certification bodies doesn't come from those who oppose it but from those who are certified & don't show traces of having gained anything from it. I have consulted at least for a few organizations here in India who hire these certified testers. Not a single test case document that these certified testers have produced has anything related to what they might have learned from the certification. 


There is a scripted test design technique that is a best practice and the most widely used one although it doesn't have a name - copy paste a sentence of requirement document into expected results column and then write the test steps accordingly. This is the most successful trick ever invented in testing. 


As a side note: Do you play any outdoor sport? Have you taken a long break from it and then went back to it? You might have been good at it at some point but when you get back after a break, you no longer feel the same comfort you had. If I were to speak to Indian testers alone for a moment, how does it feel to hold a cricket bat and face a fast bowler after taking a break from cricket? 


Bits & Pieces


I admire ants. They have a way to deal with problems. They don't say, "Oh my God! That cake is about 1000 times bigger than my size. How can I eat it?". They break the huge cake into bits that they can process, carry it home and then come back for the next bit. Doing it bit by bit helps them to achieve the goal of moving the entire cake into their colony.


While testing a product whose domain / technology that I don't know, I try to remember the ants. I learn one bit, use that bit to frame tests. The tests I perform with the bit I learned, help me learn about more bits of the system and I choose to eat bit by bit or byte by byte if my mouth has got bigger.


I may know nothing about a system when I start but at the end I can learn so much about it that it would amaze me or my audience if I tell what I discovered going bit by bit. I demonstrated this in one of the workshops where I learnt the user base of the website, traced the kind of users, identified why those users might be coming to the site, what kind of problems such users might be facing, what kind of tests to be run, what could be the most important problems that is making the audience to not give more sales + found 14 issues + 10 questions to the developers - all in one hour. I had enough feedback to the development team that they got busy working on a few things that gave me the time I wanted to go learn what I wanted about the product.


Live oracles


What are the business analysts doing? Are your sales and marketing folks so busy that they don't have time for you? How much do you interact with them? Have you invited them for a paired testing? Have you talked to them about the importance of they being with you for an hour in a month while you are testing?


If that's one set of questions, here is another set: Why in the world do you want to know everything about a domain when you know its not possible for any human being to do that?


Fooled by foolishness


Some organizations who hire testers only because of their domain knowledge, are fooled in other ways such as, those testers probably know little about testing to be called as testers. Its opportunity cost. I have spent all my time trying to be a better tester, you ask me to troubleshoot a network router, I may end up testing it or learning about it by testing it. Ask a network admin to test a router, she may end up troubleshooting or reconfiguring it.


Some organizations who continue to think that domain knowledge is the most important skill for a tester to even apply or be interviewed by them are blind to the fact that most issues that their customers are reporting has probably got nothing to do with having testers with extreme domain knowledge in the team.


There are tons of testing problems these organizations are not paying attention to, while they see their problems are because they don't have enough testers with enough domain knowledge.

Everyone of us choose to be not fooled by something and that exposes us to be fooled by something else.

The domain particle


Do you have a domain particle that makes you think "domain knowledge" is the important thing for a tester to perform well? 


I don't want to take it out. I want it to be there so that I can help you mutate it. If you could help yourself mutate it, and make the domain "learning" instead of "BFSI", "Telecom", "Multimedia", "Whatever"... I think you would have cracked what you are likely to in 20 years from now. That's an opportunity to be wise without needing to age for that.


Did you say that?


Those who speak about exploratory testing are asked, "Are you saying there is no value for scripted testing?". Those who speak about using brains to test are asked, "Are you saying there is no value in automating tests?". Those who speak about Check Automation are asked, "Are you saying checks are not tests?" while the speaker/author didn't mean any of that. Those who speak against certification are asked, "Are you saying people shouldn't get certified at all?". Those who speak about wasteful documentation are asked, "Are you saying documenting is a bad idea?". Reading some of the above paragraphs, some of you might have had a question, "Are you saying all ISTQB/CSTE testers don't know test design?"


So, whatever anyone says, there is a group of testers aggressively waiting to ask such questions. They are not doing anything wrong. They are just being themselves. Some representatives of the group are likely to ask me, "Are you saying domain knowledge is not needed to test the product?" and I am going to punish them by asking them to re-read this big post.


BTW, I am looking to hire testers of Banking / Web Application domain testers. Please send me your resume as soon as possible. 


Also, if you are a tester from Hyderabad and want to meet Rahul Verma, Dhanshekar and yours truly on 27th evening, email me. We wont mutate your particles! We will be attending Google Test Automation Conference in Hyderabad. 


Post bio for my self reference: Took 5 days to write the first draft, changed the style and contents 3 times, 1 external review, 3 self review, 5 hours of editing and finally publishing it. Published from my cousin's place in Delhi. Laptop: IBM Thinkpad

What is our competition on?

There is a trend that is picking up in the software testing industry. To be more precise, I am talking about a trend of competitions, prizes and rewards. This is good. I love the idea of being able to compete globally and with testers whom I may not have known otherwise.

When someone wins a testing challenge and the report is published to those who participated, it influences others to ape some of the good practices and end up creating their own styles out of it. The flip side, what the winner does becomes best practice for wannabe winners of the next competition. For instance, having been practicing video recording as a part of my bug reporting style, I posted videos of bugs in Utest bug battle last year. When I coach testers, I also help them understand what kind of bugs need video recording and help them practice it. Santhosh Tuppad who is a regular bug battle winner in Utest told me that many testers have started to upload videos instead of screenshots and he published an experience report where he mentioned that point.

At this point of my post, I'd like to appreciate all companies putting up competitions that is helping to bring out new, fresh and different kind of talent to the public view. I like to support you in as many ways I can. I have been a participant, creator, winner and loser of such competitions. Its all fine.


However, I am starting to have some concerns about the competitions that are coming up which demands voting by public to decide who wins. Now, I need to clarify something. I am not talking against those companies who are doing it. As a matter of fact, I am not talking against "anything". The company wants a way to get more people to know about them and such voting based competitions help in doing that. If I start an organization, I'd like to do things that helps in getting a lot more people to know about the services my organization offers. 

The problem with the voting to decide a winner, could be hurting, to those who have put in lot of efforts but couldn't gather enough votes. For instance, Eurostar conferences organized a Videostar competition. By looking at the marketing flyer of the competition which said something like, "Put on your Holywood director hat" and about creativity, I assumed the video with the Holywood movie type creativity will probably get me to be the Videostar and did the Joker act. Anne Marie did a video that I personally liked. Nothing less of all that was from Rob Lambert who had a different way of putting things that he wanted to talk. Maybe all that was driven by seeing how creative were Eurostar folks who put up this video. Finally, the video that won, by that I mean, the one that got the most number of votes, was not so exciting as others in the list. We are independent consultants who don't have a mailer list to whom we can send and help generate a lot of votes for ourselves. Being an employee of a large company and gathering votes is much easier, especially if the Head of Testing is the one asking for votes. I guess that's what happened. Now, I am not questioning about anyone's ability but I am talking about the system.

I could have still won. I could have got a 99% lead over all others if I wished to. The Videostar page recorded a vote from a browser-computer and registers it or probably sets a cookie as well so that I cant vote twice. However, I could clear the cookies, refresh the page and vote for me again. How long would it take to automate this and leave it running overnight to wake up in the morning to declare myself as a winner?

I didn't win the Videostar. That is a testimonial that I didn't try winning with the "hacked way" I discovered. Oh, if you think there would have been an assessment of votes coming in from the same IP, I could have gone a step ahead and used the cloud to make it up or use tools that helps me mask my IP and hence not reveal where the votes came in from.

In that case, should winning be decided by a panel of judges who may not understand my skills? For instance if I take up the ISTQB Foundation Level exam, I will fail. I did fail in a mock test that I took online. Does that mean, I don't know how to test or I don't know the foundation of software testing? 

A competition has a set of rules. I am just asking if the rules can be made in such a way that the skill factor plays a vital role in winning and not the votes or my ability to memorize answers and vomit it out on an exam.

Let me repeat this: I am fine with companies organizing competitions to increase their visibility but I want to know if there can be a better way to do it than the voting system. I am dreaming of testing competitions where the winner is judged based on the skill demonstrated. A report is published by the panel as to why the winning entry amongst others were eligible for the top prize. Every step we put must help us move forward. Movement is important but there is a vector in it. What direction are we moving in and by what magnitude?

Bug battles are one kind of a competition that I like. There are more kinds of skills that companies can try to focus on. When I organized a testing challenge for Test Republic, I did a Bug Advocacy Challenge. I think there needs to Bug Investigation Challenges, Rapid Test Planning Challenges, Test of agility challenges, Test Management Challenges, Interviewing Tester Challenges, Collaborating with Developers Challenge, Understanding Requirements Challenge that can help in bringing out winners with skills of different kinds. 

So, there was a competition announced from Eurostar on blogging. I was about to pounce on it because I wanted to go to Eurostar and I have been one of the earliest Eurostar blogger. I skimmed through to see if there was anything related to voting and there it was. I decided to not enter the competition. 

I would be highly stupid if I was trying to write this post to create a negative impression of Eurostar or any other organization. That is not my intention. I'd like to say that Eurostar is a great conference I want to go but not because I got a lot of votes than someone else. I want to go if my skills get me there. I understand why Eurostar or companies that are running a voting based competition might be doing these competitions but I'd like them to think if there can be a better way to do things. I have respect for organizations like Utest and Eurostar because they are trying to do some work that is helping people change things the way they do. That is the reason I want them to be able to cause a much higher and better influence on the community, to take it forward. 

A tip to the winner

If you, a tester, happen to win any testing competition, here is something that I hope you think about from what I say to myself from my experience of winning several testing competitions; I won because the mightiest chose not to compete. This thought helps you a lot when you work with people. They shall embrace you not see you as a person they should stand away or just merely stare at.

Competitions could remain but what is our competition on? 

• Choice A.  Number of votes
• Choice B. the breadth and depth of skills? 

To add a little humor to this serious post; what is your vote for? A or B?

Finding Nemo Answers: Shmuel, William & Bangladesh Testers

So, I posted a challenge for software testers in December through this post : Why testers need to learn to write code that works. It was widely tweeted, circulated and even translated to Russian.

At the end of the post was an exe file with a challenge in it. I was practicing coding and wanted to do it in a way that helps me and others. I probably thought many Indian testers might pounce on it but that didn't happen. Even the testers I mentor didn't seem to take it serious. A part of mentoring is to not get disappointed when your mentees are not excited about a thing you are excited or you want them to be excited. 

However, there were some pleasant surprises. Thanks a lot to Alexei Barantsev, Shmuel Gershon, William Fisher & Sajjadul Hakkim and his team at Bangladesh.

I am going to be presenting a review of Shmuel Gershon's and William Fisher's work. What about Bangladeshi testers work, then? Well, its in encrypted form which I will link in the bottom.

At this point, I'd like to warn you that if you plan to attempt the challenge, you might not want to read further. However, if you don't want to attempt but read the answers and review, please go ahead. Think for a moment, maybe two and see if you really want to read the answer before attempting it. If you don't know what the puzzle is then the following wouldn't make sense to you, so just play with this and come back.

So, you decided to read their answers. No problem, you still made a wise choice :)

If you want to read Shmuel Gershon's experience report and answer first before reading my review on it, please do so. Here is the link to his blog.

The Black Box Tester Approach!

Shmuel, "So I downloaded Pradeep’s application, and got to work! Rather than just trying to solve the puzzle, I looked at it as if the mission was: “This is commercial 'roulette' style game. Stakeholders want to know it the game logic can be broken/learnt, which could mean a substantial loss of money when people start winning every time.”

That's a nice start. He didn't see it just as a puzzle but set a bigger mission to himself and made it more interesting. The way he set the mission didn't contradict a bit to the mission I had provided yet was different from the actual. That's a cool thing to do in such contexts. I would have been more curious to jump in to the puzzle and treat the mission as it is. So, that's a learning for me.

• 1. First, I ordered my environment to allow efficient and organized work:
  
  
  
  
  
  1. Increased the Screen Buffer Size of the Command line I was using to 500 (it turned out that I would do well on less, even half, than that (see point 2.b). But it didn’t hurt).
  2. Renamed the executable to be shorter and without number version: ren fnemo_1.7.6.exe fnemo.exe
  3. Changed the prompt to something shorter, cleaner, and that gave me context about this task against the other Cmd lines windows open: prompt $Cfnemo$F$S$G

Testing Hint: Organizing your testing environment before you start will likely help you during your tests. After you’re in the heat of fight, it will be harder to stop and get organized.

Shmuel provides a testing hint to all of us that is important but not practiced often. I should say that I would jumped into it without bothering to do any setup because

• I tried some basic executions of the program, to grasp the feeling of what it does and how hard it is to find Nemo. This also taught me what are the messages returned by the application when Nemo is found, and when Nemo isn’t found. They’re “Found Nemo this time!, Nemo Gill Bubbles SharkTooth Flow Phamplet Stinger” and “Ah! bad luck, didnt find Nemo this time!” respectively.
1. I also learnt that Pradeep used Perl and Perl2Exe to do the app.
2. Additionally, the application clears the screen at every execution, so the big command line buffer is not so helpful.

While trying to developer test this program, I found that not clearing the command line left over was a little bit irritating to me to perform my next test and hence I added a clear screen in the code. You might be interested to note that I clear screen at start because when my code erred, I didn't want to see it. A clear screen gave me a fresh look at the output and I was happy. It is possible that other programmers do something similar which impacts the usability of the product.

So, programmers aren't wrong unless they don't want their code to be tested.

The next time I write code, I shall keep this in mind and would try giving the user an option if he'd like to clear the screen every time this program is launched. That might solve one problem, probably without creating others.

Sarmila commented on the Challenge page about disassembling the executable. Although by learning all the assembly one can learn about the rules that move the fishes around, it would be extremely difficult and overkill. But this made me think on how Perl2Exe works… Maybe it just wraps the Perl interpreter and the Perl script together? If so, what if the script is stored internally in clear? I tried to open the app in an Hex editor, but no, the Perl script isn’t in clear text there, it is obfuscated.

• I also tried the useful Strings tool by SysInternals. My intention was to see if, maybe, the array of fishes could be seen at the app code, in order to learn the initial order of fishes. It couldn’t.
• That ended my cheating session  , from here on I used only a black box functional approach.

So, Shmuel came to a conclusion that the code might be obfuscated. Well, its not. Watch out for William Fisher's approach! However, Shmuel did try to cheat. When the programmers say, "Don't test this part!", that doesn't mean you don't do any tests there. A little bit of cheating could reveal bugs that remain even after the feature is moved to a testing ready state. Maybe, you might learn something about the software or maybe learn how the programmer stages his or her work. That insight is helpful, too. So, did Shmuel cracked without cheating?

I don't know, why not read further?

By now, I knew that Nemo changed places between plays.

1. Nemo also change places between invalid parameters (which may or not be a bug).
2. Moreover, the other fishes change places too, even their placement related to Nemos’ placement changes.
3. So I tried to see if Nemo will return to the same place in any consistent way or number of times. Running that is easy, you just enter “3″, “3″, “3″… and count where Nemo was found and where he wasn’t.
• I discovered there is regularity (for “3″, its every 2, 3, 17, 5, 10, 5…), but the regularity was irregular enough, and also changed for different positions. This was likely a consequence of the real logic, rather than the logic itself. This correlates to the movement of fishes, but does not cause it.

Shmuel starts with simple tests and that's very important. I don't know why our brains try to move towards complicating things than simplifying it. It looks like "simple thinking" is a skill and a must for tester. Puzzles, especially those which we were unable to solve were the ones which we complicated it so much that we put a puzzle2 over the puzzle1 and tried solving puzzle2 and yet tried getting the answer for puzzle1 through that. Solve it directly!

Another detail that helped me in the game was knowing the developer and the purpose of the challenge. This puts a lot of content in the context. Pradeep is a rather playful  and would put some tricks into it.

1. First I tried to see if Nemo could sometimes be in any placement bigger than 7 (there are 7 fishes, places 1 to 7, but there were no rules as to where the fishes were limited to be).
2. I tried in a toss of up to 220 attempts, and Nemo wasn’t in place 8 even once. So I let this idea on StandBy for now.
3. Then I tried… what if the “Minimum attempts” input at the beginning of the app affected where Nemo appears? 
   
   
   
   
   
   • 10 is the default for the “Minimum Attempts“. Nemo is at place 1 at the beginning there.
   • 11 attempts chosen: Nemo is at place 1 at the beginning too. Maybe it does not affect?
   • 12 attempts chosen: Nemo is at place 4! Bingo!
   • 13, 14, 15, 16 showed that the original places were cycling at ‘3′ intervals. That means that when trying to find Nemo on the default 10 attempts, it will be in the same placements as 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52…

Having understood some bit of me, Shmuel made a conjecture that I could be playful and do some tricks to fool you all. That's a kind of thing like trying to understand the psychology of the programmer and model tests around it. 

You also notice the OFAT heuristic? You don't know OFAT, no problem, its One Factor At a Time. Now, I don't need to explain MFAT, right? Shmuel, tried using the OFAT heuristic (although he might call it in a different name) to narrow down or learn about the behavior. It did work well for him. He found something valuable that looks like helped him proceed on solving the puzzle.

You’re seeing that “220 attempts” above, and thinking if I really entered each number manually.

1. By this time, I decided that I was in need of some automation to insert inputs to the application.
2. The original app had no apparent automation capabilities, so I decided to use the internal input redirection of the Command line: 
   
   
   
   
   
   • By using the “<” operator, I could redirect an input file “input.txt” to the app using “fnemo.exe <  input.txt“
   • I extended that to “fnemo.exe < input.txt > output.txt“, which gives us with a complete dump of results that can be searched with Notepad.
3. We had an easy automation now, and this can help our sapient testing. I was exploring, and using automation to help me try different things help me explore my possibilities and get a bigger picture of the situation when needed.

Look at that powerful stuff. A tester shouldn't be classified as a manual only tester or an automation only tester but the industry designations suck. Shmuel provided evidence that it is the context in which either of the approach is used. Well, manual isn't a good work, so I agree with Sapient tester, as James Bach coined. Testing is related to human brain not the hands that type the input. All good testers I come across, don't separate manual and automation testing. They see it as approaches that compliment each other.

However, lets see what Shmuel did after that?

Do you still want to continue reading? 

---

• With the knowledge and tool I had so far, I did a lot of trials in order to find where Nemo is at each iteration. This could give me a hint if he moved in any regular or consistent manner.
• So I made a big file of winning moves (a lot of trial and error went in it).
• It was now time to model the results:
• First attempts at modeling the results:
• I got this: Nemo’s position: 1, 3, 5, 3, 2, 7, 3, 4, 7, 4, 4, 1, 5, 5, 2, 5, 6… 
  
  
  
  
  
  • Nemo was certainly jumping around without any clear regularity, even when I did it twice the times there was no repetition

On the Second modeling attempt, I tried to see if a more visual model could help:

1	2	3	4	5	6	7
X
					X
				X
		X
					X

Even in a table with many more results, it made no sense.

"No answer is also a useful answer". The first time I said that was in one of Elizabeth Hendrickson's exercise that I volunteered to guinea pig. Shmuel did fine. He looked for a pattern in the output but didn't get it and that was a clue for him that there appears to be no pattern. I wish he had done more tests on it but then sometimes critical thinking is a trap.

I didn't want someone to find the pattern of where Nemo is, that easily. I set the trap.

I changed my approach. Looking only at Nemo was not giving enough insight. Plus, I knew the other fishes moved along, so there might be a hint in their placement.

• I got this table:
• 
  
  
  
  
  

  Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
  Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow
  SharkTooth	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles
  Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles	Flow
  Stinger	Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet
  Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth	Nemo
  Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth	Flow
  Flow	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles
  Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger	Nemo
  Flow	Stinger	SharkTooth	Nemo	Phamplet	Gill	Bubbles
  Flow	Phamplet	Stinger	Nemo	Gill	Bubbles	SharkTooth
  Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
  SharkTooth	Flow	Phamplet	Stinger	Nemo	Gill	Bubbles

• That didn’t look good. Nemo moved spuriously, and the other fishes either jumped aimlessly or stayed at their place. Nothing made much sense.

Shmuel, while he reads might be surprised to know that he didn't have to do this table. The code does the table, I had enabled logging. Shmuel didn't probably pay attention to his C:\ folder :)

Must admit that version 1.4 didn't have the logging but Michael Bolton, yes, the singer and tester, did a Rapid Test and suggested this feature. Maybe Shmueld did extract this from the log file but his report doesn't suggest he did.

I still thought that looking at the other fishes will bring the breakthrough needed.

1. In order to learn the movement of the fishes, I did a table of their placement in relation to Nemo at every run. 
   
   
   
   
   
   1. i. This is what I got:
   2. 
      
      
      
      
      

      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
      Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
      Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger
      Nemo	Phamplet	Gill	Bubbles	Flow	Stinger	SharkTooth
      Nemo	Gill	Bubbles	SharkTooth	Flow	Phamplet	Stinger

   • See anything interesting? The rows repeat themselves alternatingly! So the fishes were not really moving around, they were following Nemo!
   • My take is that Pradeep has two different arrays: One for Odd rows, the other for Even rows.
   • This may not be true, but it doesn’t matter for us. When testing, we not always know what exactly the programmer wrote in the code, but we infer a mental model. If it suits the needs, it is a good model even when not the real thing.
   • Many times I think these fake mental models are even better than the real thing. It’s the best way for them to act as an intuitive Oracle when analyzing an application.

"Boy, you got closer". That was my expression when I read the above. Good going pal!

So, if you have got curious to see what he did after that, you must visit his blog. I am not going to be revealing it further but to spoil your curiosity, he did crack it.

Kudos Shmuel! I hope other testers congratulate this effort of yours. It was very inspiring. I wish I could say "You could have done this and that" but it doesn't make sense to say that especially after your brilliant effort plus you did crack it! Your excel sheet, shows the way!

Now to William Fisher!

Whitebox Testing Approach

Aren't you curious to see the whitebox approach in action?

William Fisher...

K





Key:









 ! = Bug
* = Action taken
? = Question to investigate
@ = Assumption
Thoughts before executing:
? Size of Array
Unsure if the size of the array
@ At minimum, it must contain 7 elements.
? "All fish change positions"
@ Do all fish REALLY change position?
Wonder if their position oscillate between a set number of actual configurations
Are the ordinal positions simply incremeted? decremented?
Randomization?
Seeded?
? Are their intial positions consistent between session?
! I understand the rules, but how do I USE this?
? Assuming that I'm to enter a number representing an ordinal position
zero-based?
 
 
Notice, he has a different style of reporting than Shmuel, yet it is equally very useful and understandable. Finally, what matters is; is the information provided in a useful and understandable manner to the target audience? If you meet that through any kind of reporting style, good job done. William appears to set the context right to read his report further and when I ask questions to myself if I do that, I don't. However, just because I don't do it and I see other tester doing it is not a plausible reason that I should do it. Knowing different ways is of help when I'd have to switch style from one kind of audience to another.
 
If I have held you till this moment, It is a big achievement. Not just mine but of Shmuel & William, too.Test 1: 
 
 
Track the fish at ordinal position 1 across successive attempts
> Minimum attempts is set to 10 but you may increase it: 0
> 10 attempts left
! I entered a number and received the message "10 attempts left". Was I only supposed to press enter?
---
> Guess the position of Nemo :0
> You probably didn't give any any input or your input was invalid;please provide a valid input
> Guess the position of Nemo :"
! Did that count as an attempt? How many attempts do I have left?
@ If the language uses zero-based, potential exists for off-by-one error if input is not decremented properly
---
 
 
I told you. I did tell you, its hard for a tester to avoid finding a bug :). You also notice he is asking questions that are different from the ones Shmuel asked. He might also have suspected if making an invalid entry or just hitting the enter might cause a movement of Nemo but did it happen? 
 
> Guess the position of Nemo :1
> Ah! bad luck, didnt find Nemo this time!
>
> 9 attempt(s) left"
! There is an extra carriage return (prior to '9 attempt(s) left'. Should that be there?
! Attempts are decrementing from 10. Opening screen says I can increase it. How do I increase the number of attempts?
@ Different code used to display max number of attempts, than what is used to display the remaining number of attempts
@ The use of '(s)' in 'attempt(s)' indicates that the code does not change output based on remaining attempts.
@ Refactoring possiblity: use one function to display the remaining number of attempts. This will accomplish the same thing
---
 
  
So, as and when tests are performed, William is thinking about constructing the code in mind. It is definitely a useful approach, at least to me. I always try to imagine how the code behind this GUI look whenever I am not exposed to it. Modeling the code, no matter how wrong, is a useful approach. I get test ideas different than what I had by modeling the code behind the GUI. Now, if there ain't a fancy name for it, don't worry. It doesn't matter.
 
 


> Guess the position of Nemo :1
> Ah! bad luck, didnt find Nemo this time!
>
> 8 attempt(s) left
End session:
Found Nemo with 1 attempt left
Having found Nemo, I've been given the opportunity to guess one more time.
It closes after 5 seconds; cannot mark the entire session.
? Does this program generate any artifacts?
? When an attempt is not accepted, do the array positions change anyway?
================
Test 2:
Manual run the same inputs in s new session.
Run session with FILEMON attached
End session:
No artifacts generated by the executable
Did not find Nemo in this session
? Is Nemo's position time-based?
================

 
 
This is cool. Suspecting a specific thing to be happening and basing tests on it and not being biased when seeing the output. Appears like a critical thinking approach to me.
 
 

Test 3:
Manual run the same inputs in s new session.
Run session with TSEARCH attached
In TSEARCH, HexDump shows some secrets:
- print "\nHave you reverse engineered the logic? If you haven't go on!\n";
. else. {print "Ah! bad luck, didnt find Nemo this time!\n";.
End session:
Need to dump ASCII Strings from exe file.
 
You expect me to talk about tools now? Right. Tools are to help testers. This is another example of sapient testing, I would say. Until we use a tool, we don't know if its going to be of much help. If you observe what William was doing, he was constantly worried about code and was also concerned about which tool could be of help to him. Tried Filemon and then TSEARCH. I didn't know about TSEARCH till William sent me the report. Wow! You got to do something to get a tester to talk to you. That's all education happens as a side effect.
Every software offers us enough clues to help solve the problem. It ain't an intentional design, I guess. I still like to think that way although I might be wrong. I am not going to say this to my client while testing for them or we wouldn't have time to talk about these.
 
And you'd want to know what happened after William got that clue? Watch out for the video.
 

If you were by any chance curious while reading this, I too am, for a different reason. I want to know what would happen when William Fisher discovers the things that Shmuel did and vice versa. I bet they'd appreciate each others effort and approach and would learn from each other. 
 
Shmuel had almost concluded that there appears to be no way to look into the array by opening the exe in the HEX editor but William's experiments violated that.

 
Both of them almost missed the log file. William discovered the existence of the logfile by looking to the code. The next time I should just pass on the .pl file instead of EXE, I guess. All beans spilled, ah.

 
 
What's the deal with Bangladeshi Testers?

 
 
Ever since I discovered SQABD.com , I am wanting to go there. I want to go talk to those testers there. Not got a chance yet. I am hoping it doesn't remain a dream. Sajjadul is a tester with thoughts that are widely respected within and outside of Bangladesh. He and his team took time out and discussed about solving this puzzle, after experimenting on it for a while. 



Watch the video: (Some part in Bangladeshi and rest in English)

Want to get your hands on Finding Nemo 2? Watch out for it in March.