Planning for a ransomware attack - an enterprise 'to do' list for self-protection

Every so often, I look at restart/recovery and hacking incidents to update what software users could/should do about them. A periodic review is useful as threats change, IT deployment technologies evolve and legal issues continue to morph.

The first virus/malware software I ever encountered was in the 1970s at college. The mainframe ‘cookie monster’ was fairly benign, all things considered, as all it wanted was for an operator to type in the word ‘cookie’ and it would just go away. That’s not the case with today’s malware.

I recently tracked an enterprise software ransomware attack over the last six months and also looked at some cloud software contracts. Herewith are some sobering observations and recommendations for software users and vendors.

Typical chronology of events

A typical ‘security event’ often surfaces when: a malicious actor notifies the customer or vendor that malware has been installed on their system; a user’s data is encrypted or destroyed; the system begins to behave in an inappropriate or incorrect manner; and/or, sensitive data is now being posted on the Internet.

The vendor or customer is expected to either comply with the financial demand (often by hard to trace offshore accounts and/or cryptocurrencies) or rebuild their systems. Not all malware is designed to trigger an economic payoff for the malcontent. In some cases, the malware is designed to disrupt an entities’ business and/or steal sensitive data (e.g., key intellectual property). Files may or may not be corrupted. Sensitive information (e.g., employee payroll, tax ID and other personal information) may be the real target and it may get sold on the dark web.

This Spring, Palo Alto Networks noted:

Ransomware groups are growing bolder and ransom demands are on the rise.

According to the 2022 Unit 42 Ransomware Threat Report, the average ransom demand in cases we handled climbed 144% to $2.2 million in 2021. At the same time, there was an 85% increase in the number of victims who had their names and other details posted publicly on dark web 'leak sites'.

A recent, sobering and must-read MIT Sloan Management Review article on ransomware had this comment:

In a recent study of 300 companies, 64% revealed that they had experienced a ransomware attack within the previous 12 months, and a staggering 83% of those paid the ransom. On average, only 8% of organizations that paid up recovered all of their data, while 63% got about half of it back.

Are cloud application customers really prepared?

I’ve seen companies buy all kinds of cloud software in the last two decades and many approach cloud security (and malware risk management) with a cursory checklist mentality (e.g., do they meet current security certification standards?). These customers ‘trust’ that the vendor will have great security for their cloud computing environment. Yes, major cloud solution providers have a lot at stake (financially and reputational) and they usually offer well-protected solutions.

But the landscape continues to shift. And, the threats vary based on whether it’s on-premises or cloud solutions. According to Data Center Knowledge:

Ransomware was targeted more at on-prem infrastructure, as was malware that set up backdoors for attackers or connected to command-and-control infrastructure.

In the cloud, ransomware was involved in only 5 percent of incidents, roughly the same as last year.

On premises, however, ransomware jumped from 8 percent of incidents to 33 percent.

Customer contract naivete

I’m not convinced that all cloud application software buyers are as savvy as they could be when it comes to acquiring new application software. First of all, customers often sign the vendor’s contracts/papers not their own. That can be problematic as the customer is examining what the vendor wants in a deal and not fighting for the things the customer wants or needs.

Indemnification and liability

In most breaches, some person or their sloppiness is a major contributor to the causal event. For example, malware or a hacker got into a system because:

• Someone used a really obvious password (e.g., 1,2,3,4)
• Someone failed to disable a former employee’s access to the system
• Key security or other programs were not patched in a timely fashion
• An employee used an unpatched, remote personal computer to access the systems
• An employee accessed questionable websites or fell prey to a phishing or other social engineering ploy
• Etc.

And don’t forget that some of the most damaging events have been due to disaffected or former employees.

Why is this important? From a contractual perspective, a vendor will not want to be responsible for the malicious or accidental acts of your own employees. Furthermore, don’t be surprised that the language in their subscription with your firm may state that:

• You have to help them mitigate total harm/damage to the site and data
• They are only responsible for getting the systems back up. Any rekeying of data is your responsibility. Any reprocessing of transactions is also on your dime. They might even want to charge your firm a pretty penny just to reset your transaction databases and master files back to a known, safe time.
• They aren’t responsible for other damages like those due to lost revenues (while their order entry system was down), reputational damage to your firm, etc. even if it was due to their own negligence.
• They might expect you to share the recovery costs and/or make you and them responsible for indemnifying each other for your respective losses.

Mutual indemnification clauses make me twitch. Why? In this scenario, some gigantic software company wants my little ol’ company to pay their legal, software recovery, etc. fees and they’ll pay mine. I don’t think so. Worse, some vendors expect you to share in the pain when the problem could have been caused by a vendor’s employee, a user of another company, or, some random hacker with no affiliation to your firm.

Who is responsible for what?

The subscription agreement will lay out the specific responsibilities and remedies that the customer and vendor will have. It can be highly restrictive and limiting. But, and this is key, it only outlines the liabilities and cures for these two parties. Other parties, like employees, suppliers, etc. are not signatories to this cloud software deal. And, since those entities were not signatories to the subscription service, these individuals can litigate if their paychecks are inaccurate, their personal information is leaked, their identity is compromised, etc.

There could other parties involved, too. The presence of open source and/or commercial systems software products may indicate that a third-party may be partially culpable. For example, if a third-party software provider failed to patch a critical component in a timely fashion, they could be liable for damages. A subscription agreement may be silent on this possibility or it may insist that your firm assist the cloud vendor in its attempt to garner damages from this third-party vendor.

As to the software customer, they are ultimately responsible for paying their employees and suppliers, closing their books, etc. The software contract might explicitly preclude the software customer from collecting damages from the vendor in the event of a hack.

Your firm should definitely have its counsel review and revise these liability contractual matters.

Data recovery

Customers are often stunned to learn how little capability actually exists to recover prior data and transactions. One vendor I know only makes your online transactions available as a CSV file dump. If you wanted to rebuild your implementation, you would need to reprocess all of your historical transactions, in order, to get the system back to some last backup point. You don’t get to backup the actual databases and their contents. This is a key point. As a customer, you don’t get to know the database schema, the names of your databases, or anything else that might permit you, the customer, to have an image copy of your data. Nope!

So, when a major failure occurs, you will either need to reprocess all of those millions of historical transactions all over again or beg the vendor to reload your information via one of their backups from a specific point in time.  Vendors will discuss how they possess a failover capability in case something catastrophic happens to one of their data centers. But, those failover capabilities may not help if some malcontent has been messing with your data for an extended period of time.

There’s one more challenge to backups/recovery: shared tenancy. Some cloud solutions have multiple customers’ data in a single transaction database or server. This type of configuration makes it difficult for a vendor to quiesce the other customers’ activity while the vendor resets the affected customer’s data. This isn’t an issue in container-based solutions.

Beyond failover protections

Failover capability is important when you consider the damage that fire, earthquake, electric failure and other disasters can trigger. But, smart customers should understand what their options are when they can’t:

• Process orders
• Pay employees
• Produce accounting reports
• Pay vendors
• Schedule production orders
• Etc.

In a recent ransomware event, customers were left to create their own alternate processes/procedures when their cloud solution went dark. Some of those customers resorted to:

• Paper, forms and pencils to capture employee time
• Re-running the prior month’s transaction as these should be ‘close’ to those for this month and that corrections would be entered in a future cycle

In that particular event, many customers were offline for 3-5 weeks.  It’s noteworthy that customers had no alternative system available to them.

Unfortunately, the service interruption wasn’t the only issue. The Personally Identifiable Information (PII) of scores of employees of these cloud customers was placed on the Internet. Depending on your cloud agreement, the vendor may only be liable for providing a credit monitoring service for a specified time frame for the affected employees, customers or suppliers.

Litigation costs/limitations

Your cloud contract may limit your firm’s ability to sue the vendor even if the breach was their fault. All your firm might be entitled to are some service credits, credit monitoring and a few other services.

The big litigation issue may not be between your firm and the cloud vendor though. Instead, employees, customers and suppliers may file all kinds of lawsuits with some seeking class action status for damages they may have suffered. These plaintiffs may sue both your firm and the software vendor. This is why you need to try to hold the vendor liable for these damages and not your firm.

In a recent ransomware event, three class action lawsuits were filed in a manner of weeks after the event occurred. All of them were by adversely affected employees.

Other costs

Software customers may have to pay a ransom and this may or may not be covered by your company’s cyber insurance policy. This is a major contracting point as the size of these payments can be significant and few contracts may spell out who is on the hook to pay such a ransom or whether a ransom will be paid at all. For example, your firm may want the vendor to pay the ransom and get the system back online asap while the vendor may not. How does this get resolved?

Software customers might want to take a really hard look at their cyber insurance policy to verify exactly what it will and won’t cover. Your firm might need to purchase additional supplemental insurance to provide more complete coverage. In particular, your firm should have insurance to protect itself from claims arising from the theft/misuse of PII and to make the firm whole should it experience significant business interruption. And, of course, the insurance should pay for the time and expenses of security experts, litigators, investigators and others needed to identify the source of the event, mitigate its impact and restore business operations.

The time problem

Few contracts I’ve seen address what happens if the vendor cannot restore operations after a defined period of time. This is quite important. If a key service, like Payroll, is down for a day or two, an employer can likely survive that. But, if the service is down for a month or more, then a business’ longevity is threatened.

Likewise, not all systems are affected by time issues equally. For example, every minute your order entry system is down could cost your firm millions in sales that it might not ever get back. On the other hand, if your Fixed Asset Accounting system is out of commission for a couple of weeks, that could be highly survivable. Overall, systems that interact with external parties are likely the most sensitive to time-intensive interruptions. Make sure your agreements reflect this.

What customers should do annually

At the end of the day, security is the responsibility of both the customer and vendor. Every software customer should have a great security consultancy on retainer. Hopefully, you’ll never have a breach or cyberattack that requires one of these firms to do a full forensic investigation. But, the time to line up one of these firms is well before a breach or attack occurs. Time is of the essence to stem one’s losses.

That same consultancy should probe/test your systems annually, at a minimum. They should look for unpatched systems, lightly protected entry points, poor integrations, etc. and recommend new capabilities to further ‘harden the target’.

That’s great for your on-premises environment but your cloud applications may need review as well. Cloud application vendors often make their recent EDP and security audit results available to customers but they don’t share everything. In fact, they may not permit your firm or its representatives to see their data center(s). Why? The less the outside world knows of their technology, the easier for them to protect it. Nonetheless, your firm should make sure any audit results, certifications, etc. are current and relevant each year.

Ideally, the monitoring of these systems is a continuous, not annual, occurrence.

Notification

The best cloud vendors have a well-considered communication plan that can be implemented immediately upon becoming aware of a breach/attack. Many aspects of the communications are documented well in advance and can be quickly tailored to the current situation.

What I’ve noticed in recent attacks is that vendor communication can be significant in the early days of the attack and then dramatically scarce after just a couple of weeks. While I don’t have any concrete proof on this, it appears that vendor communication wanes as soon as litigation rears its head. I suspect this happens when lawyers don’t want the firm to disclose anything that might show (or expand) a firm’s culpability whether it’s real or not. Regardless, both the customer and the vendor need their own communication plans.

An adjunct to the communication issue is the need for the vendor and customer to notify and work with law enforcement. This should be obvious and immediate.

What’s often not addressed is how the affected parties will interact with the press/media. A crisis PR firm might offer the best guidance on this matter.

Expect competitor & analyst reaction

Some financial analysts may see a malware/breach as a reason to downgrade the prospects of the affected firms and as an opportunity for those unaffected by it. In a recent incident, that’s exactly what happened when some financial analysts saw one vendor’s breach as a sales/switching opportunity for the vendor’s chief competitors.  To date, I have not heard of any actual deal or customer attrition occurring though.

Sudden changes in market share maybe doubtful although the attack could be a factor of sorts in some later deals. At a minimum, an affected vendor (and all other vendors in the same space) can expect tougher questioning in RFPs, RFIs, selections and renewals re: cloud security, security audits, threat detection and more. This type of event rarely just affects one vendor as it makes all customers and prospects (re-)examine their solutions.

The other reasons why you’ll likely see few changes in market share after some malware/ransomware events are that:

• Software selection and implementation projects take time and cost money. Pulling together a team, getting the project green-lit, getting this project to the top of a full IT project slate, etc. are just some of the hurdles that must be overcome to get these efforts going. And, since so many firms have previously automated the affected function one or more times in the past, the net-new benefits that could be accrued from such a deal are dwarfed by the net-new costs. Unless the customer is feeling especially vulnerable, it’s unlikely there will much activity here.
• Moving from one solution to another can also be risky. Unlike horseshoes or hand grenades, ‘close enough’ won’t work for implementing new solutions. These projects are thoughtful, considered efforts that are rarely taken lightly.

Your 'to-dos'

 

1. Don’t sign any cloud (private or public) software subscription until you get a top cloud contracts attorney to review the contract especially the liability and indemnification clauses.
2. Ensure you have a workable backup solution should a cloud solution fail for any period of time. You may need a paper-based solution, an old on-premises solution or something from the vendor. You might be able to shift your processing needs to another similar solution in use by your firm at one of its other locations. However, this approach could prove expensive as it may trigger increases in user counts or other contractual items.
3. Get clarity around who pays a ransom, if anyone does, and how much say your firm will have in deciding how the vendor deals with ransom makers
4. Assess your exposure and risks with embedded private label or white label third-party products
5. If your public cloud applications vendor offers you a private cloud option, make sure you are aware of and comfortable with the different security responsibilities, requirements and liability risks with each deployment option.
6. Get a great predefined disaster recovery plan – not a simple failover plan. These are different items altogether and your firm should have an ability to recover in mere minutes (not days, weeks or months later).
7. Model several different kinds of threats and determine what specifically your firm and the vendor are responsible for, when certain tasks must be completed, who pays for each activity, etc.