Does a Real Incident Count as a BCP/DR Test?

During a conversation with a fellow professional, I was asked whether the operations incidents that affect business performance would qualify as having tested the company's BCP/DR plan.

My answer? Submarine movies.

Some of us as kids watched the old black and white World War II submarine movies, like Run Silent, Run Deep with Clark Gable or Destination Tokyo with Cary Grant. Other people remember the more recent movies, like Das Boat, Crimson Tide or Red October.

A common announcement by the captain is, "Dive! Dive! Dive!" when an emergency (ie the enemy) appears. The crews are ready for anything and work with speed. Everything seems automatic and according to plan.

What can be missed in these movies is how the submarine crew gets to the point of rapidly and efficiently responding to the emergencies: Drills, drills, and more drills. Many submarine movies show a ship's officer with a stopwatch or timer, drilling the crew on how to respond to commands to prepare for emergencies. Often, the crew isn't ready, so they drill again, and again, until the captain is satisfied.

In our professional lives, I'll bet that we are not crew on a WWII submarine that needs to decisively respond to an emergency. What we do respond to are operational issues with server, network, cloud, and software failures. And then there are the critical times when malicious attacks by software or hackers threaten our business environment. Recently, the Crowdstrike event was an "all hands on deck" moment for many businesses. If not directly affected by the Crowdstrike outage, businesses noticeably felt the impact of cloud environments like Azure as it felt the impact. What can you even do when these situations hit?

One of the ways that businesses can prepare for an operational and business impact is through their BCP/DR and incident response plans (you do have these plans, don't you?). These plans are the "Dive! Dive! Dive!" response to impactful issues where participants have a part to play in communication, decision making, who is involved, etc. But an organization can't flawlessly execute to these plans without. . .you guessed it. . .drills (or should I say “tests”)!

At least an annual test of the incident response and BCP/DR plans is a way to become familiar with what to do and whether plans are the right fit for people and potential situations. The tests focus on the effectiveness of plan components, using a real-life scenario for the underlying context.

But back to whether an actual incident that significantly affects business operations can be considered a test of the BCP/DR plan. The answer is "sort of, but no", and here's why:

An incident response might be a good test of a specific situation, but might fall apart in a different situation.

The focus of a real-life response is to the situation at hand, not focus on whether the BCP/DR plan itself is complete and works. The old adage, "You can't serve two masters" applies.

The purpose of testing an incident response or BCP/DR plan is to decide what works and doesn't, making the plan the "go to" response. An incident might highlight what works and doesn't, and even introduce new approaches, but how many times will you hear, "Gee, we should update our plans with this new response"? Not many times. The focus is on the fix and not updating the process. Few will be taking notes when in the midst of the incident.

Certifications like ISO-27001 and SOC 2 require the official action of reviewing response plans, including the when, who, and what that was reviewed and approved for the next year. An incident by itself does not accomplish that purpose.

The execution of a planned, practiced, and tested response plan results in the actions that happen when the real emergency takes place. An operational incident is a great way to contribute lessons learned from incident response and BCP/DR plans, but should not be the driver. The goal should be to execute to a plan and then adjust the plan based on what's learned during its execution.

So, an operational incident response should not count as the only test of a BCP/DR plan. It takes reviewing, testing, and being familiar with the plan so it can be executed effectively, with feedback to improve the plan. So, "Drill, drill, drill!" (or should I say, "Test, test, test!") until the response is seamless!