Ransomware incident response is a technical problem right?

Editor's note: This article is by Sudarshan Pisupati, Principal Security Research Engineer, Zscaler

In my experience helping clients respond to ransomware outbreaks, the amount of planning and preparation is inversely proportional to the damage incurred due to an unfortunate ransomware outbreak. Where organizations stumble is when, in the heat of the moment, they succumb to decision paralysis, waste time and money, and scramble to respond logically as an attack plays out inflicting untold damage. When a breach occurs, unnecessary trouble begins because security teams treat outbreaks as a purely technical problem when ransomware incident response is actually a project management problem.

The average cost of ransomware unpreparedness: $260,000

In early 2021, we categorized and tabulated the total time we spent on five separate ransomware response activities in the first 96 hours after the outbreak.

In each scenario, the victimized organization enlisted us (Smokescreen) as an advisor to the CXO to help make good decisions quickly. In all cases, we were on all-day Zoom calls with various teams. Each call had, on average, 25 stakeholders representing organizational departments, vendors, partners, and consultants. Therefore, each hour spent in the investigation resulted in the usage of 25 person-hours. In four of the five incidents, business connectivity was restored after approximately nine days.
Here are the key problems we identified:

Loss of access to core on-premise security and connectivity infrastructure had the single biggest impact on response efforts. In three of the incidents, services like email, Active Directory, VPN, and SIEM were all knocked offline. Investigation barely progressed in the first 24 hours of the ransomware attack, resulting in the loss of nearly 600 person-hours across the average call attendance of 25 stakeholders.
Lack of communication between various teams resulted in tremendous person-hours of time wasted. Often teams had insufficient direction, duplicated work, and withheld information. They lacked timely access to important data and wasted time on activities that had little impact on immediate business objectives. This piece delivered the second biggest cost, at approximately 20 hours of lost time per stakeholder, yielding 500 person-hours without meaningful progress towards bringing business back online.
Decision-making paralysis resulted in the loss of approximately 200 person-hours. Some tough calls have to be made when faced with ransomware. Information is always incomplete in an outbreak scenario. Quick decision-making could have saved roughly 200 person-hours (at approximately eight hours lost per stakeholder). Instead, incident response teams wasted time by delaying information requests, consulting too many voices, and waiting for more information that would have had little impact on final decisions.
Lack of understanding of ransomware threats: Approximately 50 person-hours were spent debating ransomware concepts and engaging in conversations and back-and-forth on topics that were not well understood by all stakeholders. A big chunk of this time was spent debating Active Directory containment measures.

While there were more categories of waste, we found that organizations waste an average of 700 hours before any meaningful progress towards the larger goals of recovery, containment, and eradication. That’s 17.5 person-weeks lost considering a 40-hour workweek.

In cases where core on-premise infrastructure was hit, this loss stands at the loss of 32.5 person-weeks. If we approximate the conservative blended cost of all stakeholders (including consultants) at $200/hour, that’s $260,000 that could have been saved simply by being prepared.

What takes CXOs by surprise during an investigation

Now that we know the estimated cost of simply being unprepared, nevermind the associated costs of a complete ransomware lifecycle and the potential ransom amount itself, the way to approach an incident as a project management concern is to factor in the following operational pitfalls in your planning:

Everything takes time

Yes. Everything takes time. Logs are voluminous. Backups are measured in terabytes. Investigation and recovery will take time. Don’t let the constant repetition of this phrase irk you.

No one is giving you the perfect, right answer

In the first week of an attack, everything is dealt with in hypotheticals. Sadly, even decisions have to be made without 100% certainty. You’ll hear phrases like, “We cannot be sure because the attacker could be elsewhere,” and “We cannot say that with certainty because we have no logs”. This is normal. Delegate technical decision-making to a trusted expert.

There are no logs and no access to security infrastructure

When you isolate your network, everyone loses access. This includes on-premise security infrastructure, which has all the logs. Sometimes attackers simply delete the data. Sometimes, you’ll find a process failed and you did not collect the logs you thought you were collecting. It happens.

Your last audit didn’t flag anything

The time for root cause analysis on questions like this is later, once the dust has settled. We have been on many calls where tempers ran high when this topic was broached. Don’t let it become a topic of discussion in crunch time. But definitely, follow up when the dust settles.

Too many stakeholders

Imagine being on a Zoom call where 20 people are giving you disparate pieces of information and it’s your job to stitch it together into something actionable. Ideally, all CXOs should talk to no more than three direct reports during the response process.

The number of impacted systems keep increasing

“Yesterday, you said 70 systems were impacted, today you are telling me it’s 150. Why is this happening? Are there more? Is the ransomware still spreading?” The impact can only be gauged as more systems are investigated. Increasing numbers are completely normal and are why defining an impacted systems threshold can help you make swift, decisive decisions.

Hopefully, you’ll never have to experience a ransomware outbreak response because you’ve taken steps to best prevent such cyberattacks. Nonetheless, by having a game plan and being prepared in the unfortunate event that ransomware hits your organization, you can avoid decision paralysis, save time and money, protect your reputation, and get the business back on track.