Credit Suisse CIO talks costs, clouds, challenges
- Summary:
- "Fundamentally we've had to redesign and challenge the way we've run distributed infrastructure for the past 15 years. Not a single thing that we've done in the past is now acceptable for the future."
"Fundamentally we've had to redesign and challenge the way we've run distributed infrastructure for the past 15 years. Not a single thing that we've done in the past is now acceptable for the future."
It's a stark declaration from Steve Hilton, Chief Information Officer, Technology Infrastructure Services at Credit Suisse Group, responsible for all global technology infrastructure.
Hilton has been speaking at the VMworld conference in San Francisco this week about the challenges he faces as an IT decision maker of a major enterprise. His account makes for insightful reading - and covers a set of challenges all too familiar to his peer group.
Crédit Suisse, like so many organisations in the financial services industry, has undergone massive transformation over the past few years with cost as the top priority throughout the journey. But at the same time, the demand for infrastructure remains what Hilton characterises as "insatiable":
"So the key is really efficiency. It's not really cost cutting. I don't think you can cost cut an infrastructure and still deliver value to the business. So that generally [it] is all around efficiency, how do we round what we've got more efficiently and cheaper?"
Virtual history
Plus ça change etc. Credit Suisse's first foray into virtualisation came four or five years ago, but it was introduced to tackle a different problem, recalls Hilton:
"The biggest problem we had was data center capacity. Data centers are growing at 1.2 megawatt a year globally, probably hundreds and hundreds and hundreds of millions every two or three years in physical data centers. Our focus there was very much how do we virtualize and get the capital down?"
In this respect, Credit Suisse has been successful, going from a CapEx spend five years ago of around $120 million per annum on physical servers down to the mid-20s.
The firm is also shrinking its power usage by 200 kilowatts a year rather than growing it. But challenges remain, explains Hilton:
"It's great we got CapEx down. But if you look at my total spend, which is now over $1 billion, over a third of it is people cost. So it's great that we've got data center and physical stuff down to less than a third. Outsourced vendor maintenance starts about a third, but it's the people. How do I do more work with less people."
"I've asked my technologists, my SAs, my DBAs to think completely differently on how they run the environment.
"We have over 7,000 applications that we support, most of them written in-house. Some of those date back to architectures from the mid-'90s. So you do have this big legacy we have to support."
Lifecycle approach
Credit Suisse takes what Hilton calls a lifecycle approach to upgrades and technology refreshes:
"Our view is if you're going to make an improvement in infrastructure, do it once, do it right and then do it for next 3 to 5 years. So we have these every 18 months to 2-year cycles of infrastructure investment, and then we bleed it and run it for 3 or 4 years.
"The concept of our platform is we provide an integrated environment, where everything from storage up to monitoring capacity performance, even change management processes, are wrapped around this bundle of compute, and we cycle that every 2 or 3 years."
The approach does throw up some startling statistics:
"Two years ago now, we spent more on new storage than we did on new compute. And this year, we spent more on new network ports than we do on new compute. That might be right or wrong. I think that's wrong because I fundamentally think that a compute cycle is what adds value to IT. A transaction comes in and combine something else spits out an answer.
"That's why our focus is now, I wouldn't say we fix compute, [but] we're now about 62% virtualized and about 85/90% of net new services going in are virtualized. I wouldn't say we're done, but my focus as a CIO now is focusing on network, storage and around the people."
Complications
Hilton regards the current cost of running Credit Suisse's network as very high, but acknowledges that for a firm like his there are complications that need to be addressed that impact on this:
"We're a Swiss bank. Data privacy is everything for us. Just in Switzerland, I have 26 different physical network zones with different levels of firewall authentication and role sets to try and protect data from different degrees. So historically, we've relied on the network to protect the data.
"I can't tell you how much that doesn't work from an operational perspective. We're the first ever customer ever to hit the theoretical limit on the number of roles you can have on the firewall. [I'm] not very proud of that, but that just shows you every single system access was a firewall change."
"I was saying to somebody just the other day, I actually believe that we need to start viewing the data center network as untrusted. We've just got to the point we're viewing our LAN, our office network, we're viewing as untrusted.
"So you say it's dirty. You then put encryption, authentication, VPN on there, and you say it's like being at home. I think we need to start guessing out with the data center because the second you've done that, you're encrypting and protecting the server asset and the data."
This in turn leads to the question of hybrid clouds he adds, and some more surprising conclusions:
"I'm forever being challenged by my developers saying, 'I want to go to the cloud'. So we go, 'Fine'. We smile. We do a project. We say, 'Let's go look at it'. Every time we benchmark - and we benchmarked just 2 or 3 months ago - we are cheaper than Amazon internally. We are.
"I've got 22,000 servers, total of 55,000 images, 40 petabytes of storage, et cetera. I have scale. So our price point's lower than Amazon. They still want to go to Amazon."
It's necessary then to push back, argues Hilton:
"Our story is 'We have an internal cloud, come and use us'.
"Do it first internally, then look at external. There's something about the hybrid cloud that's very interesting for us because that's where you start to say, 'I vaguely trust the bit in the middle if it doesn't matter where the endpoint is'.
Unsurprisingly then perhaps, Credit Suisse currently has no public cloud instances, although there is some SaaS usage for internal applications such as HR and expensing, what Hilton calls non-business apps:
"Outside that, no, we are not looking at public cloud.
"There's a whole economics of if you've got your own capacity, why would you go somewhere else because you've got that fixed cost? That fixed cost does go, give it 5, 6, 7 years, and that fixed cost clearly goes.
"But now, again, I'm a Swiss Bank as well, please take what I say as a fairly exceptional use case, but we're not looking at it at all."
And while hybrid cloud is something that is worth considering, Hilton isn't keen on the idea of OpenStack:
"My answer's no. It's not worth it. I've got 15,000, 20,000 VDI systems running on VMware. I've got probably another 30,000 virtual servers, 25,000 to 30,000. I like having a single fabric. It's very important to have a single fabric. If I go back and go OpenStack, that's not where the money is.
"The money is in my people having to build, having to certify, and run it and operate it. The money we spend with integrated stack like VMware pays for itself, magnitudes, like hundreds of times when it comes to my people's cost to run it."
And again it comes back to cost, as it always must. Hilton argues:
"Network is by far the biggest CapEx and OpEx save over the next 3 years and we have no choice but to embrace it.
"I remember years and years ago, CIOs don't get fired for buying Cisco, that old adage here, I'll never get fired for buying IBM. I now get fired for buying IBM, because it's too expensive .
"We're about to get to the point where I honestly think that we will actually break the Cisco data center in the next 3 years. I resisted that for the past 6 years. I now think we're at a point where we'll do that."
