What can we learn from one an example of failure?
We all know that running a load test on your site is best practice. By putting demands on your systems, you can discover the true capacity of your site, as well as identify any weaknesses and bottlenecks. That’s as long as you do it right of course.
One company, sadly, didn't.
Beware the celebrity tweet
A member of the senior management team for a known brand decided they needed to run a load test on their web channel. They had good reasons – they knew a celebrity was going to be tweeting about them and expected some heavy-duty traffic as a result. So they hired a third party to run the test. So far, so forward-thinking.
But, they made one little mistake. One that added up to a colossal waste of time. They failed to let their internal IT team know what they were doing.
Maybe they wanted to see how their IT team would react under pressure – after all, it wouldn’t have been a true test if their team had bolstered their systems in preparation. Or maybe they simply forgot. Either way, it backfired on them.
Protecting the business, failing the test
When the third party ran the heavy load test and the system started to receive load alerts, the IT team did exactly what they were supposed to do. They immediately picked up the unexpected hits on the web channel and swung into action. Since they weren’t aware of any testing, it simply looked as if they were under attack. So they blacklisted the relevant IP addresses and blocked all requests from that third party.
For the IT team it was a win. They’d successfully thwarted a suspected attack and by catching and blocking the source of the hits, they continued delivering uninterrupted service to their customers.
However, in terms of the test, it was a total failure. The third party submitted a failed test report to their customer. From their perspective the site had fallen over (and done so remarkably quickly). As a result, the business learned nothing about the site’s capabilities and wasted a whole lot of money in the process. And all because they failed to communicate with their own colleagues.
So what can we learn from this story?
1) Testing is a good thing. Ambush testing is not.
Yes, you need it to be a true test. But if everyone involved doesn’t have the same clear objectives, then the test will be a waste of everyone’s time (not to mention your money).
2) Consider different scenarios for increased traffic.
There are many reasons a site might experience heavy-duty traffic, from hackers to celebrity tweets. If you have a clear idea of the possible sources, you can have a plan in place for each that doesn’t always involve blocking access.
3) Communication is everything.
Enough said.
