San Mateo County - moving away from a 'one tech per project' mindset

Martin Banks Profile picture for user mbanks July 10, 2017
Jon Walton, San Mateo County CIO,  explains how a pilot virtual desktop project fuelled a mindset shift around the county’s IT infrastructure.

Jon Walton, the CIO San Mateo County, is a man who likes a challenge. So when he left his job as CIO of the San Francisco local authority back in 2013, he moved not just from a city to a county, but also from a well-resourced IT environment serving a totally urban community of just 49 square miles, to a mix of urban-to-agricultural land covering 870 square miles.

And while serving roughly the same number of people, the county had around 25% of the IT and public service staff, and less than half the IT budget.

The workload covers the same type of government, with the same type of health care, public work, safety and security issues, but over a much larger geographic area with a wide range of demographics. Walton's challenge, therefore, has been to take some of the things being done in San Francisco - such as improving infrastructure, improving public access and providing more open data services -apply them to a different environment that he describes as not long out of recovery from the recession.

There were some changes of circumstance. Walton had 'owned' everything in San Francisco’s IT environment, such as the fiber network, robust infrastructure, and data centers, because services to facilities such as the city’s major international airport were under his remit.

When he came to San Mateo there was just one fiber pair, that ran from Redwood City to the County’s main hospital. There was also one main data center,  based on a traditional model of 'N’'racks of servers, storage and networking dedicated to specific projects.

No more 'one tech per project'

One of the first challenges he saw was a need to replace the entire infrastructure, which was then made up of a mix of systems from NetApp, EMC, HP and even a few IBM AS/400s, all acquired to meet the needs of specific IT projects.

The vision was to head to the cloud or to have more virtual systems. With that in mind, Walton met with Nutanix about three years ago and set up a pilot project in 2015. He was aware that this might carry some risk as the company was still then young, with the county as customer number 130 or thereabouts.

The pilot focused on  meeting a demand for virtual desktops around the county. Because there was no hardware decision made on the subject, it was decided to see if they could run it on the Nutanix boxes as the backbone for that project. Walton recalls:

We were still looking at doing individual projects, and at San Fransciso I had been used to having lots of memory and everything being delivered on a semi-truck. So, I was very surprised when we did the pilot project because it seemed like a very small appliance compared to what we were used to. And it was up and running in under a day.

Perhaps more to the point, he had a core IT staff of around ten people who all bought-in to the Nutanix approach. They decided to do some comparative tests against the likes of NetApp and Simplivity, before opting to expand with Nutanix. Walton says:

Though we started on the VDI side, it just grew. We knew we had an aging infrastructure for all of our other services and Nutanix just seemed to be a natural for the backbone of all of our virtualised infrastructure.

San Mateo County is a VMware shop, and servicing that was not the initial focus for the Nutanix environment. But the engineers took it on from day one because it gave them extra capacity to work with, while  its modularity made it easier to add more resources as required. So the team started out enhancing the services at the hospital and down at Redwood City where services for public safety were improved.

Nutanix also carried out a study on the Return on Investment being made on IT by the county, including such aspects as the power consumption and space saving costs that have been made possible, adds Walton:

From a CIO and financial management standpoint, that made my job easy. My boss is fiscally conservative, so whenever I can show a saving, for example where we can take 15 racks and crush them down into one rack, where there are space savings in the datacentre and we can cut the power costs, that is appealing to non-technical people. When the engineers love it from a performance and ease of management point of view that makes it a win:win as far as I am concerned.

The ease of management has been of particular benefit to the IT staff at San Mateo County, not least because the systems Nutanix has replaced consisted of a range of different systems and solutions all puchased on a project by project basis. This had meant that implementations of upgrades or additional services were often delayed. This, in turn, has often led to frustrations amongst the public the county serves, when they have not managed to access or use services, either publicised but delayed, or more commonly these days, services they feel the county should be providing.

Meanwhile IT staff would  become frustrated because they would want to install Version 'X' of an application, which had all the features needed to provide some of those desired services, but would find it difficult, if not impossible, to do so. This would often end up being established as a new project requiring new hardware, new training environments and a raft of other additional facilities.

Walton also acknowledges that the county had fallen into the common trap of allowing itself to slip two or three versions behind on some applications, particularly where the upgrades had added little of specific value to their operations. Then the latest version comes along with solutions to meet the peoples’ growing service demands, but the cost of playing version catch up has then become a problem.

Taken together, these created an environment where the old environment made developing new applications difficult, he admits:

Because the Nutanix tools like Prism are so easy to use, we could see that we had the capacity for a test/dev environment. We had the capacity without even planning for it. With the old environments we often over-built them because we were never sure whether we were going to run out memory or drive space. And once built you daren’t touch it because you don’t want to impact production.

In practice, the Nutanix environment allows the team to `borrow’ spare capacity on lightly loaded nodes to run development and test processes without affecting the production environment. This has therefore given Walton and his team the chance to start the development of additional or completely new services and the resources to start work on them.

When technology becomes commodity

According to Walton, Nutanix subscription model is also more flexible when it comes to payment terms:

The normal process is that when someone leaves an IT organisation [the project manager or other budget holder] the next person in gets the whole bill for a project. But the Nutanix subscription approach is more modular and that has allowed us to build capacity onto the back end.

So when staff come and tell him they have a project idea he is now normally in a position to ask when they will be ready to spin it up as a test site. If necessary, the County can then order up the additional production resources that might be required to maintain that back-end capacity.

One common application users want is Disaster Recovery (DR) but in Walton’s view, the Nutanix environment provides that as a something of a by-product, as VMs can be replicated without interfering with production services as and when required. This can be to wherever is thought most appropriate, from different nodes within the same datacentre, through replication to another data center in the county’s network and now, with the Nutanix partnerships with Commvault and AWS, it is able to selectively replicate relevant services up into the public cloud.

He also cites increasing levels of simplicity in using Nutanix, where most of the technological issues are now commoditised out of any discussions:

We used to have long discussion about what was the best architecture or what disk drives were the best to specify – for me it was always Western Digital. Now, I couldn’t tell what drives we are using. I think most of the vendors still see technical nuances as key product differentiators and it is important for us to understand those issues, at least to an extent, so we don’t go down a dead-end technology road. But at the end of the day it is now all about performance, up time, and cost. And we are getting four times what we got before, for about 25% of the cost.

Walton's department sometimes manages the IT functions of other departments that are not directly part of the San Mateo County Authority’s remit. For example, the county has some 22 cities, which have individual responsibility for some police and fire services. But as new projects come up for them - and have to be balanced against their own budget requirements - some of them are now being merged into the County services, particularly where issues such as public safety play a significant role.

In addition, Walton has put in fiber to connect all the county resources, wherever they are, with some of them sited in the local facilities of those cities. This has prompted the latter to start enquiring about using the fiber to connect to some of their own services running remotely back at county. This means they can also connect to an exploit county services if that fits their needs.

When people live in an area they tend to want similar services to where they work or have lived before, so next for us is to try and make our services as ubiquitous as possible.

My take

Here is a CIO starting to confront the issues that commonality of technology can present. Instead of the silo-isation of 'one technology solution for each project' where resources are contained and unsharable, the production environment can yield spare resources to support new development and testing opportunities. And in resource-strapped sectors like Local Government, this can lead towards actually meeting at least some of the expectations the people served would like to see implemented.

A grey colored placeholder image