Main content

Fighting fires - how Pure Storage helps the City of New Orleans prepare for disasters

Derek du Preez Profile picture for user ddpreez June 20, 2024
Summary:
City of New Orleans CIO has implemented a new disaster recovery plan, using Pure Storage, following a ransomware attack in 2019.

An image of the french quarter in New Orleans
(Image by llambrano from Pixabay)

With over 400,000 residents, New Orleans is the most populous city in the State of Louisiana. Known for its beautiful architecture, rich cultural heritage, and one of the best music scenes in the USA, New Orleans is also a popular tourist destination for visitors all year round. 

However, the city hasn’t been without its challenges. New Orleans suffered a crippling ransomware attack in 2019, which brought the city’s technology services to the ground, resulting in the Mayor declaring a state emergency. In addition to this, the city has been hit by 19 hurricanes in the last four years alone - again making safety and public services extremely vulnerable. 

As such, the City’s CIO, Kim LaGrue, keeps disaster recovery and operational resilience front of mind. Her team not only supports the technology services of citizens, but also those of the 5,000 city employees, which include police, EMS and public safety teams. 

The 2019 ransomware attack was a turning point for the City. Thankfully a $3 million cybersecurity insurance policy was in place to lean on and the technology team had the support of the Mayor to invest in infrastructure, but it was clear to LaGrue following the critical event that New Orleans needed a disaster recovery plan that would result in less downtime (the recovery period following the ransomware attack was months). Central to this plan was implementing Pure Storage. 

It’s worth noting that at this time there were a number of concurrent things happening that meant the City of New Orleans was ready to take new resiliency plans seriously. 

Firstly, Mayor LaToya Cantrell has played a central role. LaGrue explained: 

Most cities struggle with infrastructure refresh. In New Orleans, we were fortunate that we had a mayor who had just come into office less than 18 months before the ransomware attack and had said infrastructure was a priority. 

Whether it was the sewer infrastructure, the road infrastructure, or the technology infrastructure, she said these things need to be upgraded and maintained. So, we have been working with the administration to understand what it would take to bring our infrastructure into a good position. We had the support of the City. 

Anyone that has followed technology projects in the government sector will know that political backing is critical to success. This coincided with the collapse of a historic building in the middle of New Orleans. Whilst a disaster, it required the Mayor, the CAO and the police chief to be stationed in one of the busiest areas of the City and have access to communication platforms outside of their traditional office environments. LaGrue said: 

Your communication strategy, your telecommunications infrastructure - it had to work. So we were really, really thinking about how this modernization could happen and how we were going to make sure that the city state is supported through any type of disaster. 

Nobody had planned for a building to collapse and for the Mayor to leave her office and get stationed in the middle of one of the busiest streets in New Orleans with her team. Whether it was cellular coverage or being able to get the emails from a trailer in the middle of downtown New Orleans, we had to be thinking about those things. And that's when you really understand that infrastructure is important. 

Disaster strikes

Needless to say, infrastructure and operational resilience was already on the minds of City officials prior to the ransomware attack. LaGrue explained that quite a bit of preparation and planning had already been done - the City was focused on consolidating operating systems, doing PC upgrades, and trying to find a way to get to flash storage:

That's where Pure Storage entered and why we had a relationship with them, because we were actually considering those things at the time of our attack. 

Where we had maturity most was understanding what we had. We had really adequate inventories of what kind of information, what data we had, we had made decisions about prioritizing that data. 

However, whilst some work had been done, disaster recovery plans weren’t totally effective in responding to the ransomware attack, with regards to getting the City up and running again quickly. Federal and state agents were brought in and it was clear that whilst there had been a massive ransomware infection and the City’s systems were compromised, there hadn’t been an exfiltration of data. New Orleans wasn’t going to pay the ransom, which meant that it had to clean everything, build a new infrastructure and then migrate any santized data:

Having cyber insurance itself was a good move, a really important move, for us to have made. Having that in our back pocket made it a much easier conversation with the City’s leadership, within hours. It was one of the first questions they asked. 

Resilience isn’t something you really understand until it stops. Then you understand what the need has been. Resilience is not a word that cities hear easily because there can be so many competing priorities for it. 

And part of New Orleans’ problem was that its point of failure was far too narrow, having just a single data center. And this is where the introduction of Pure Storage in building out the new infrastructure started to play a central role. LaGrue said: 

After the cyber attack we knew we could not have all of our eggs in one basket. And actually a move to new storage, Pure, afforded us a little more - we saved about 30-40% of what we thought our costs would be to build new storage. We were able to leverage that cost to build out a secondary data 

A new test

So, the City of New Orleans’ strategy shifted from a single data center to two data centers that sat three hundred miles apart. And just two short years later, the City faced a new test in the form of Hurricane Ida. However, whilst the hurricane brought with it strong winds and rain, it wasn’t these that directly brought down the technology infrastructure. Instead, the hurricane took out the power at the primary data center, which triggered a generator that then caught on fire. 

This time, however, because of the City adopting Pure Storage’s ActiveDR, which provides global disaster recovery capability and maintains application performance with continuous replication and near-zero RPO/RTO, New Orleans was able to get back up and running in 48 hours. LaGrue said: 

Our cyber attack recovery took months. And we expected that we would have seven to 10 days [downtime], if something happened, to put that infrastructure back together. But in 2021, when we had the fire in our data center, we stood up operations in 48 hours. We did not believe that it would happen. But we followed the plan that we had made built on our Pure infrastructure, and it actually worked. It was like flipping on a light switch. 

We had immutable backups of our infrastructure right there. And so we brought in some core equipment, we moved it literally from New Orleans to Baton Rouge, connected everything and brought up that backup infrastructure. And it worked as it was designed to work. It just did. The storage, the platform, bringing up the VMs, getting access to the city's data just as we had replicated it into that environment, it all worked. We were back online in 48 hours. 

Gaining confidence

LaGrue said that the recovery plan - given that it worked - has given the New Orleans’ staff and citizens confidence that the City is funding the necessary infrastructure and can deliver in maintaining services:

The confidence that it gave our team, in themselves and in our technology solutions, was just immeasurable. 

And the success of the recovery since the fire event has meant that the City is now looking at how it can further invest in resiliency and its technology infrastructure: 

It lets us know that we have to do that across the board. We need to find ways to make all parts of our technology infrastructure resilient on the same level. And the offset of cost allows us to make those investments in places where we weren't making them before. 

Having a platform that helps us build a recovery strategy, and where we can do all this in one place, and within one solution, that saves time, saves resources. That allows us to go and tackle some more complex problems to make our infrastructure that much more resilient in other places.

In terms of advice for other organizations, particularly those in the public sector, LaGrue said that it’s probably worth technology teams carrying out cybersecurity insurance exercises. Even if they can’t afford they can’t afford the insurance premiums, which may well be likely, the exercises are thorough and allow for organizations to recognize their capability gaps. Equally, LaGrue said that communication is key: 

The communication between all parts of technology and city leadership really have to be considered. You have to be communicating with your business what the impact of continuity of operations could mean to them. 

Loading
A grey colored placeholder image