Prepaid bugs !

Hi Reader,

Prepaid bugs ? "Pradeep do people pay in advance to see a bug?"

Yes, indeed, you too have been a part of it, if you are from India. For other country chaps, alert your mobile service provider...

Did you get any clue ?

__ Prepaid bugs __

We had 2 hours break in a training that I was attending today and I got a chance to interact with a guy. Can there be a chance where I have not talked about testing to anyone ?
Hardly any...

I started off introducing myself and so did he and then we started talking about testing.

He had worked with Tata Teleservices earlier and I happened to ask him
"What was the worst phase of your career with Tata Teleservices?" and the story goes...

Sometime back the prepaid billing system which is deployed in iN ( Intelligent Network ) went for a toss for almost 3 days. This is a well reputed company and how could they have allowed such a thing to happen. You should be aware that there are certain things that goes beyond our control and so it did happen.

The team there started to diagonize the problem and after a day they found out that ... the few servers that interacted with a Avaya soft switch where not communicating with the soft switch as per the design.

As per his explanation the pre-paid billing servers communicate with the Avaya soft switch in a Time Division Multiplexing manner but they found that the Avaya soft switch was unable to take the load as all the servers were trying to ping the switch at the same time and that lead to the whole problem.

Now there are a few questions that had to be answered ...

1) How did this problem occur when it was running fine for a period of 1-2 years ?

Ans: Personnel from Tata Teleservices contacted the Avaya support team and their query being a priority one was routed to the France R&D team and then Avaya consulted their sub-contractors who also worked on the same product. Finally it was identified that it was a hardware failure which lead to mismatch of time slots to ping the switch.

2) What is the solution ?

Ans: A new hardware had to be deployed replacing the faulty one.

3) Why did it further take 2 days to get it fixed ?

Ans: The software/driver/hardware was not available in India at that point of time and someone was sent to Singapore , he then purchased and after his return they fixed it.

4) Wasn't there any back up , come on I don't believe this ? ( This was my question too )

Ans: Yes there was but it took a lot of time to move a customer info from one server to another and that itself would take a lot of time and in the mean time the problem can be fixed and also it would take more time to move from one iN to another and then retrieve it back.

Lessons learnt as a Tester -

I consider this a good learning for me and hence here is what my perception of it ...

1) Ideally Avaya should have got the hardware which failed at Tata Teleservice iN and should put that as a tool to be used for Load Testing for the soft switch they are working on.

2) With an organization outsourcing a job and the outsourced guy again outsourcing to someone and the chain goes on ... will lead to testing becoming more complex at the blackbox/system test level.

3) If it is a real-time or critical application that are installed on real-time networks, there is nothing wrong for a Tester to suggest the organization to buy back up hardware or also maintain a list of nearest Vendor/Retailer list for the products they may be in need.

4) For such a scenario, had the France Testing team asked to reproduce the issue, it would have been the toughest phase of their Testing career and my dear Testers, when you are unable to reproduce something after exhausting your tries, time to burn your hardware and then making a last try.

5) Hardware Testers should take such examples and do a research to find out ways to indicate a hardware failure. Of course there are LED displays for some systems but remember the above problem occured with something called intelligent Network.

6) Testers should ideally look around the world and that is the only way they can progress, you maybe doing some testing on some applications and the above incident may not be of any help to you but if you think you can learn a lot.

__ End of _ Prepaid Bug ! __

"Thinking and Laughing is free of cost in this world , either you think or laugh at yourself"

Regards,

Pradeep Soundararajan

pradeep.srajan@gmail.com

Disclaimer : This post is not to affect any business of Tata Teleservices or Avaya and is an effort to learn from what has happened. Also I take no responsibility if the above incident was unreal and this post is in existance as per what the person I met narrated to me. He too maynot be responsible if he had lied since it is time to appreciate his imagination. No organization I am working with control my thoughts and this post is not from a PC that an organization has provided it to me for official usage. My current ip 192.168.1.100 which belongs to BSNL broadband connection :D