Too often these days I hear this coming from senior leadership at financial institutions (FIs) we’re having conversations with: “I’ve outsourced. I don’t need to do anything towards disaster recovery.”
How wrong they are.
During the recent severe winter weather events in Texas, most of the customers that we worked with were outsourced. Employees trying to gain access to mission critical services on behalf of their FI were also personally impacted by the same event impacting the FI.
Yes, they were concerned about doing their job, but they were also concerned about no power, heat, or safe drinking water at home. And, if they didn’t happen to have everything they needed at their fingertips, going anywhere to get anything, like last night’s backups, was nearly impossible for 2-3 days.
FIs Still Need Control of Their Systems
If you peruse the writings of the federal regulators, you’ll find several references that when you boil them down, sound something like this: Some FIs elect to outsource some or all of their technology to a third-party service provider.
Although the use of third parties is allowed, these outsourcing arrangements don’t transfer the responsibility from the FI to still comply with required security measures and controls. These include disaster recovery testing and business continuity planning.
For the sake of time, I’m going to focus this article on disaster recovery (DR) testing. That said, we should always keep in mind that the regulatory guidance is written in such a way that tells us that DR is a subset of BC, and as such, the DR strategy should be in sync with the overarching BC plan.
We’ve all heard the primary benefits of outsourcing – predictable monthly costs, reduced risk, increased safety, and the ability to allow staff to spend more time on other things besides keeping the lights on in the FIs technology arena.
Those are all great benefits. The looming question here is, “If I’m outsourced, and I lose my [blank] (fill in the blank with: power, internet connectivity, backup generator, or building), how do I access my outsourced technology?”
After all, with mobile apps, internet banking, and even legacy phone-based voice response systems, your customers may have more access to their data than your employees. Who wants to operate in that kind of environment?
Have a Disaster Recovery Plan
Oftentimes, outsourcers will allow their customers to participate when the outsourcer is testing their own DR capabilities. This is a great thing, for both the customer and the outsourcer.
It helps the customer better understand the ins and outs of the outsourcer’s recovery strategy, including the recovery time and recovery point objectives (RTO and RPO) that the customer can anticipate. It also gives the outsourcer real-time feedback from their customers participating in the test. This is all part of the fiduciary duty of the outsourcer.
The Real Disaster Recovery Challenge
What about the fiduciary duty of the FI? Participating in the aforementioned exercise covers part of the FI’s responsibility, but what about the other part?
This is the part where the outsourcer is just fine, business as usual for them, but the FI is faced with answering and dealing with my question from earlier, “I lost my [blank], now what?”
Mentioning Texas again, in many cases it wasn’t just one thing, it was multiple things. This is where annual testing of the FI’s DR strategy comes into play.
Ask yourself these questions:
How will I access my data if…
- I don’t have power?
- I don’t have internet connectivity?
- I no longer have a building?
While there are a lot of options, the place to start is with your outsourcer. What strategies or solutions do they offer to meet your needs in any of those situations?
While the pandemic has likely enhanced your ability to work from home or work remotely, the above questions still need to be asked.
Build a Process
As you think about everything that needs to be tested, a common misperception that I hear is, “We don’t have time to test all of that in one day.”
You know what? You don’t have to! As you parse through the writings of the regulators, the emphasis is on comprehensive testing annually. It doesn’t say test everything within the same eight-hour period.
A friend of mine in the airline industry was explaining to me the approach that most of the major commercial airlines take. When a plane goes offline for maintenance, they don’t go nose to tail, once a year, fixing or doing preventive maintenance. That plane is going to go offline multiple times during the year, and have different areas serviced each time.
But, within their window, over the course of several months, every aspect of that plane has been reviewed, tested, and either fixed or replaced.
DR testing for FIs can be handled the same way. You just need to ensure that over the course of the window (in our case a calendar year), all of the necessary components get reviewed, tested, and if necessary, fixed.
Work the Disaster Recovery Not the Budget
One of the more concerning trends I’ve seen over the past year or so, is deploying something new into production during this annual budget cycle, and then deploying the DR for it during the next annual budget cycle. While some have blamed the pandemic for this, I’m not sure it is. It may make the bottom line look better, but you’ve just significantly increased your risk profile for at least a year related to whatever you just spent all of that money on rolling out! Does that really make sense?
During your next IT Staff meeting, take 15 minutes and ask these questions:
- What’s our DR strategy for enabling employee access to outsourced systems, no matter what happens?
- Do we have a DR strategy in place for anything that isn’t outsourced?
- For 1 & 2, has all of it been tested during the previous calendar year?
- Do we have documentation to show our auditors and examiners what our plan is and when each component of it was tested?
In the end, you need to know that no matter what happens, you can continue to serve your communities, especially at a time where they may desperately need your services.
While the Texas scenario was a weather event, accurately forecasted several days in advance, much like a hurricane scenario, this felt very different from the FIs’ perspectives. People are prepared for hurricanes, or big “normal” expected weather related disasters, and they practice for it. People didn’t seem to be prepared for this or know what to do as they attempted to navigate their FI’s recovery.
“Business is about people. It’s good to remember that and treat the people you deal with as individuals. RELATIONSHIPS keep us together, and together we can do great things.” ~ Jack Henry & Jerry Hall
Remember, your FI is about the people employed there, and the people they serve. As leaders, both within the FI and within the community, people look to you to lead, and lead well.
Don’t risk your FI, that place where you and your people rely on for their income, their sense of well-being, their opportunities for community involvement and support because “I’ve outsourced. I don’t need to do anything towards disaster recovery.”
.svg)
