When someone wins a testing challenge and the report is published to those who participated, it influences others to ape some of the good practices and end up creating their own styles out of it. The flip side, what the winner does becomes best practice for wannabe winners of the next competition. For instance, having been practicing video recording as a part of my bug reporting style, I posted videos of bugs in Utest bug battle last year. When I coach testers, I also help them understand what kind of bugs need video recording and help them practice it. Santhosh Tuppad who is a regular bug battle winner in Utest told me that many testers have started to upload videos instead of screenshots and he published an experience report where he mentioned that point.
At this point of my post, I'd like to appreciate all companies putting up competitions that is helping to bring out new, fresh and different kind of talent to the public view. I like to support you in as many ways I can. I have been a participant, creator, winner and loser of such competitions. Its all fine.
However, I am starting to have some concerns about the competitions that are coming up which demands voting by public to decide who wins. Now, I need to clarify something. I am not talking against those companies who are doing it. As a matter of fact, I am not talking against "anything". The company wants a way to get more people to know about them and such voting based competitions help in doing that. If I start an organization, I'd like to do things that helps in getting a lot more people to know about the services my organization offers.
The problem with the voting to decide a winner, could be hurting, to those who have put in lot of efforts but couldn't gather enough votes. For instance, Eurostar conferences organized a Videostar competition. By looking at the marketing flyer of the competition which said something like, "Put on your Holywood director hat" and about creativity, I assumed the video with the Holywood movie type creativity will probably get me to be the Videostar and did the Joker act. Anne Marie did a video that I personally liked. Nothing less of all that was from Rob Lambert who had a different way of putting things that he wanted to talk. Maybe all that was driven by seeing how creative were Eurostar folks who put up this video. Finally, the video that won, by that I mean, the one that got the most number of votes, was not so exciting as others in the list. We are independent consultants who don't have a mailer list to whom we can send and help generate a lot of votes for ourselves. Being an employee of a large company and gathering votes is much easier, especially if the Head of Testing is the one asking for votes. I guess that's what happened. Now, I am not questioning about anyone's ability but I am talking about the system.
I could have still won. I could have got a 99% lead over all others if I wished to. The Videostar page recorded a vote from a browser-computer and registers it or probably sets a cookie as well so that I cant vote twice. However, I could clear the cookies, refresh the page and vote for me again. How long would it take to automate this and leave it running overnight to wake up in the morning to declare myself as a winner?
I didn't win the Videostar. That is a testimonial that I didn't try winning with the "hacked way" I discovered. Oh, if you think there would have been an assessment of votes coming in from the same IP, I could have gone a step ahead and used the cloud to make it up or use tools that helps me mask my IP and hence not reveal where the votes came in from.
In that case, should winning be decided by a panel of judges who may not understand my skills? For instance if I take up the ISTQB Foundation Level exam, I will fail. I did fail in a mock test that I took online. Does that mean, I don't know how to test or I don't know the foundation of software testing?
A competition has a set of rules. I am just asking if the rules can be made in such a way that the skill factor plays a vital role in winning and not the votes or my ability to memorize answers and vomit it out on an exam.
Let me repeat this: I am fine with companies organizing competitions to increase their visibility but I want to know if there can be a better way to do it than the voting system. I am dreaming of testing competitions where the winner is judged based on the skill demonstrated. A report is published by the panel as to why the winning entry amongst others were eligible for the top prize. Every step we put must help us move forward. Movement is important but there is a vector in it. What direction are we moving in and by what magnitude?
Bug battles are one kind of a competition that I like. There are more kinds of skills that companies can try to focus on. When I organized a testing challenge for Test Republic, I did a Bug Advocacy Challenge. I think there needs to Bug Investigation Challenges, Rapid Test Planning Challenges, Test of agility challenges, Test Management Challenges, Interviewing Tester Challenges, Collaborating with Developers Challenge, Understanding Requirements Challenge that can help in bringing out winners with skills of different kinds.
So, there was a competition announced from Eurostar on blogging. I was about to pounce on it because I wanted to go to Eurostar and I have been one of the earliest Eurostar blogger. I skimmed through to see if there was anything related to voting and there it was. I decided to not enter the competition.
I would be highly stupid if I was trying to write this post to create a negative impression of Eurostar or any other organization. That is not my intention. I'd like to say that Eurostar is a great conference I want to go but not because I got a lot of votes than someone else. I want to go if my skills get me there. I understand why Eurostar or companies that are running a voting based competition might be doing these competitions but I'd like them to think if there can be a better way to do things. I have respect for organizations like Utest and Eurostar because they are trying to do some work that is helping people change things the way they do. That is the reason I want them to be able to cause a much higher and better influence on the community, to take it forward.
A tip to the winner
If you, a tester, happen to win any testing competition, here is something that I hope you think about from what I say to myself from my experience of winning several testing competitions; I won because the mightiest chose not to compete. This thought helps you a lot when you work with people. They shall embrace you not see you as a person they should stand away or just merely stare at.
Competitions could remain but what is our competition on?
- Choice A. Number of votes
- Choice B. the breadth and depth of skills?
To add a little humor to this serious post; what is your vote for? A or B?
Exactly for the same reason i refrained from participating in the Bug of the month contest from utest.
ReplyDeleteI like the list of challenges for testers that you had mentioned..Really a good list -
"Bug Investigation Challenges, Rapid Test Planning Challenges, Test of agility challenges, Test Management Challenges, Interviewing Tester Challenges, Collaborating with Developers Challenge, Understanding Requirements Challenge"
Such competitions will lead us/testing community to learn and move forward in right directions
My vote goes for C, that is peer reviewing. For the XP DAys Germany there is a peer review process in place. Each participate can review session proposals up to a deadline. After the deadline the conference panel selects those sessions which seem to fit in based on the feedback from the participants, i.e. what they want to see. This is wonderful, and this works. And overcomes most of the fallacies you mentioned.
ReplyDeletePradeep,
ReplyDeleteWhen I saw the videostar competition I was a little surprised about the voting set-up - BTW, I voted for you, Rob & Anne-Marie using 3 different browsers (others had twittered about this possibility) - I thought at the time that this is open to abuse. This doesn't mean the winner wasn't worthy - the lack of transparency raises questions about the competition format rather than the entrants.
When I saw the blogstar announcement - I read it carefully, paying attention to the voting, at the time it said something like "the most popular" blogger would win. This triggered me to write a post - both positive to the idea of generating interest and encouraging some bloggers (yourself included) to enter, but also questioning the voting format.
About the same time there was a discussion on the writing-about-testing list raising similar questions. A couple of days later the competition organisers changed the voting format. We'll see how this works... I don't have the time to enter it this time around - but hopefully, if the competition gets 'good reviews' both for content and format then it'll be repeated and more bloggers will enter.
I too would like to see more competitions like the Test Republic Bug Advocacy challenge (I thought there would be more) - when I won that I felt very humble and thought that although I may have shone at that point it doesn't automatically mean I will shine tomorrow (the Black Swan effect). I enjoyed that challenge and learnt a lot (I wrote an experience report about my learnings).
Seeing more of these competitions as well would be good for the community - but they do also have a cost in terms of time and effort to the judges - public voting is 'easier'. Ooops ...
Pradeep,
ReplyDeleteWhen I saw the videostar competition I wondered about the format of the voting - I voted for you, Rob & Anne-Marie using different browsers (got the tip from twitter). So then I thought - where's the transparency of the format?
When I saw the blogstar announcement I read the format - it said something like the "most popular blogger" would win, but nothing about how that would be determined.
At the time I wrote a post - both positive about generating interest and thinking about some testers that might enter (including you) - but also with a question mark over the voting format.
About the same time there was a similar discussion on the writing-about-testing list.
A couple of days later the competition organisers clarified how the winner would be decided.
I don't have the time for this competition this year - but if it gets good reviews (including its transparency) then hopefully it will be repeated.
I would also like to see more competitions like the Test Republic Bug Advocacy challenge. When I won this I thought - ok I've shone on that day, but it doesn't mean I will shine tomorrow (the black swan effect) and I must work at it.
The competition was a learning exercise for me (I wrote an experience report about it), which is one of the best types of competition for a tester IMO. Hopefully, there will be more like that and the ones you suggested.
But having these types of competitions means more work (effort and time) for the judges which is why a public vote is maybe 'easier'. Ooops...
While winner is selected/chosen by voting system, it shouldn't be called as 'Competition' rather it should be called as 'Election'.
ReplyDeleteAccording to Oxford dictionary:
- Competition means "the activity or condition of striving to gain or win something by defeating or establishing superiority over others."
- Election means "a formal and organized choice by vote of a person for a political office or other position"
Here noway to relate voting system with competition. So, it's upto Company/Organization to decide whether they are going to organize a competition or something else.
-Selim
Hi Pradeep,
ReplyDeletegood post. As a kind of support, let me tell you, that i had similiar thoughts when VideoSTAR was happening.
A) Open to fraud, if not properly secured.
B) Cant compete with the masses (of collegues/employees called to vote).
But to be honest, i would do the same and organise my peeps to vote for me. :-)
Maybe they should only allow votes from participants, aka people who had uploaded a video for VideoSTAR.
Give them 2 votes each, so they can vote for themself and for another, which they like.
I entered TeamSTAR, cause it *is* a chance to win tickets for EuroSTAR (which, as a medium company, we never would spent that much money on) and i have the hope, that it will be different.
The voting&winning description reads far better, in my eyes, this time.
As for competitions and/or awards .. i think it caters to the human nature and it can help to improve the individual and therefore the society, *if* done in a proper way.
It should keep a balance between challenging the individual to give his best and a cutthroat mentality on the other side.
Cheers,
Maik
Hi Pradeep,
ReplyDeleteA very good point. We are trying something different with BlogSTAR and TeamSTAR.
With BlogSTAR, we have appointed an independent committee from the test community to judge entries and decide on their best 3 Bloggers from the competition.
The test community will then get one day to vote for their favourite from the 3. The voting process this time has also changed - you must now validate your email to vote (I'll drop you a mail to show you what I mean)
TeamSTAR will go to the vote in much the same way as VideoSTAR but using this new system.
The EuroSTAR Team
Very true about the VideoStar competition. Now, it is so easy to clear the cookies automatically and it can be even automated while I am playing video game and votes keep getting counted. And also, it is not the votes but the creativity behind the video should also be seen.
ReplyDeleteSimilar with uTest Bug of the Month where the award goes to for likes. I did not like the idea because here the competition is with the number of friends in Facebook or the quality of the bug as the title reads "Bug Of The Month".
Cheers,
Santhosh Shivanand Tuppad
Hi Pradeep,
ReplyDeleteFor one of our internal QA events, we were thinking of introducing a vote-driven format for selection of papers but reverted back to review-panel based selection, because we can not rely on authenticity of votes. A candidate from a "bigger team" would get undue benefit.
In external competitions, the same problem would exist. Along with count of votes, sources of votes must be taken into account.
There's another aspect to votes. Votes are counted not weighed. For example in our popular music reality shows, what matters is "public vote" whether it comes from a music expert or a person who has never listened to music in his/her life.
@Pradeep, my vote goes to "B".
ReplyDelete@EuroSTAR - Email validation does not stop "an employee of a large company" (and gathering votes is much easier, especially if the Head of Testing is the one asking for votes) from winning. I am guessing every employee of that company will have a unique email id.
Also there are free email id providers (including Gmail) where a bug (or a feature) exists. Try this step - send mails to
1. myemailid@gmail.com
2. my.emailid@gmail.com
3. my.email.id@gmail.com
4. m.y.e.m.a.i.l.i.d@gmail.com
All email will go to same id myemailid@gmail.com or any of the above four ids with which you logged into gmail.com
Gmail currently ignores the .(dot) in the email id. I can create AVeryLongEmailIDforVoting@gmail.com and vote as many times as I can create different ids out of this id using variours .(dot) positions. All validation link will come to same emaild id and I can validate each one easily. How will you stop this?
@Markus,
ReplyDeleteThanks for the info. I liked that approach.
@Simon Morley,
I am sorry to have missed reading that post. Indeed you did write out on this topic much before I did and I should have linked to your post. I didn't notice it. Now, I have updated the blog post with your link.
@Maik,
I wish your team the success for TeamSTAR. Eurostar seems to have taken the next step in securing things work fine.
@Eurostar,
Thanks for being open to ideas.
hi Pradeep..
ReplyDeletei m looking for interview questions on testing...will u mind suggesting me some website for it or if u have any lot of interview question..please post it...and thanks for clearing all myths about testing..