HSBC online banking outage shows its lack of cloud smarts

How long should an online service outage last? For UK business customers of global banking giant HSBC, online banking was inaccessible the whole of yesterday and has rapidly become so again today. The bank finally put the service back online yesterday evening but it has been overwhelmed again as traffic rose today. One of those customers is diginomica, which has given me, as the company's finance guy, a user's-eye view of the outage. It's been an uninspiring experience that shows a surprising lack of cloud smarts on the part of HSBC.

I've been using Web-based online services to run my business for almost two decades now — my first online business venture, founded in 1998, ran on (what we would now call) a cloud CMS and a cluster of ancillary cloud services. All that time, I've used online business banking, and in fairness outages have been rare.

But when banking services fail, they have a habit of failing for extended periods, which is not what I've come to expect from other cloud services.

I learned very early that, when an online service goes down (or when your Internet connection fades), you simply go off and do something else for a while. Often, a few minutes is enough. In rare cases, you may have to leave it for an hour or two. (If it's your Internet connection, then switch to your backup — which can be as simple as using your mobile or going to your local coffee shop). Unless it's something that's really mission critical every minute of the day (in which case you should have a fallback in place) such outages are an irritating inconvenience but not disastrous.

Where it gets serious is if the outage extends for more than half a day. No self-respecting web service provider can afford such lengthy outages, and if they do occur, it's essential to make full disclosure of what's going on. By the standards of leading cloud service providers, HSBC has performed astonishingly poorly.

It gets worse

What really surprised me as the day wore on yesterday was the clear lack of preparedness at HBSC for such an outage. I first attempted to log on around 9am and found the service responding too slowly for me to login successfully. No worries at that point, but I did have some payments I had planned to process so I returned several times during the morning. Each time, the website was responding so slowly that, even if you did get as far as logging in, it would take so long that your session would get timed out. On a couple of occasions I did manage to log in but could not get any account details to appear on screen.

By late morning I was wondering how widespread the problem was. A quick check on Twitter and Google News revealed that it wasn't just me. Mobile and online banking were locking out most individual and business customers of HSBC throughout Britain.

At lunchtime, things got really bad as for a short while I was getting a 404 error page when I tried to access the online service. Then at last the bank put up the above notice informing customers that the service was offline.

After restoring service yesterday evening, the bank's online services failed again this morning as customers come back in droves to catch up on what they had hoped to get done yesterday. I managed to complete a couple of tasks shortly after 9am during a grindingly slow session which then dumped me out to the above notice while in the middle of attempting a transaction. There's been no sign of any improvement since.

HSBC made a statement yesterday stating that the problems had not been caused by a denial of service attack, which implies that the cause was an internal error of some kind in the bank's systems.

My take on HSBC's failings

Although very little information has been coming out of HSBC, here's my assessment of what's been happening and how it should have been handled better.

• A weekend upgrade has gone wrong. It's fine to choose a slow holiday weekend to change datacenters, put in new hardware or upgrade the database. But clearly HSBC did whatever it did without a proper fallback plan if things went wrong — and missed something vital that it should have caught during testing.
• Insufficient disaster planning. It was a mistake to keep the service running all morning yesterday instead of pulling the plug immediately to work properly on a resolution. I was pleased when I saw the 404 message at lunchtime because at least it meant they had switched stuff off and were putting something new in place. Less pleased though when I discovered it was a 'come back later' notice but at least this took the pressure off the failing systems instead of remaining in panic mode.
• Lack of transparency. The cardinal rule is to keep customers informed, including as much detail as possible about what went wrong and what is being done to resolve the problem. HSBC has utterly failed that test, with its social media accounts reduced to repeatedly saying how sorry the bank is and to try again later — and more recently, committing that customers "will not lose out" though with limited details as to how that will be achieved. Making matters worse, there seems to be limited information sharing going on inside the bank, so that the social media advice is often out-of-date compared to what customers are experiencing.
• Lack of collaborative thinking. Even though customers couldn't access their accounts all day, no one thought to amend the system that automatically sent them text messages to warn them they would face a £5 charge if their accounts remained in overdraft. This was especially annoying to customers with funds available that they could have transferred if only they'd been able to access their accounts ...
• Not enough investment in online. Although HSBC has been making some improvements to its online banking, many of these have been done on the cheap. One thing that constantly amazes me about my business accounts with HSBC is the huge quantity of printed statements I receive in the mail — the bank must spend literally hundreds of pounds every year on mailing paper to me that I would gladly opt out of if given the choice. Why couldn't some of that wasted expense be diverted into the online systems? The ROI would surely be rapid.
• Not realizing HSBC is a cloud operator. I suspect the real problem here is that HSBC's management sees online banking as an ancillary service to its main business. But its customers' reaction to the current problems is I hope helping it to realize that it is now in the cloud business and if it can't operate online services as efficiently and smartly as the likes of Amazon, Google and Facebook then its customers will go elsewhere (on this point, it's interesting today to go back and read this story from almost two years ago: HSBC innovation boss urges cloud rethink by financial services firms.
• Failing to keep up with its customers. Retail banking is changing rapidly and HSBC is going to have to be far more responsive, respectful and radical in serving its customers. Like any incumbent business, it has to change in line with what is now possible with connected digital technologies.

I've mentioned the importance of having a fallback several times in the course of this post. There was a time when banks were the fallback — their entire image was built around reliability.  After the 2008 crash and now repeated failures of online banking systems, that brand value will soon be irrevocably trashed — just when new competitive threats emerge. Still, I suppose they can argue that at least the pay's not bad.

Image credit: HSBC screenshot.