On Friday, July 19, companies ranging from airlines to hospitals to financial institutions began the day being involved with a mass global tech outage. CrowdStrike, a cybersecurity company, sent an update to their Falcon sensor program for companies with the Microsoft Windows operating system (Mac & Linux hosts were unaffected). Unfortunately, this update had some issues installing on Windows systems, and many saw the dreaded “blue screen of death”, which, through 2024, now reads on Microsoft operating systems as “Your PC ran into a problem and needs to restart. We’re just collecting some error info, and then we’ll restart for you.”
It is completely possible (and even likely) that some of you reading this were impacted by this issue in one way or another, either personally or professionally. I’m sorry you had to deal with that, and know it wasn’t your fault. It was a vendor relationship in which you trusted the update provided was fully tested before being rolled out. CrowdStrike advised they were/are “working closely with impacted customers and partners to ensure that all systems are restored” and that they’ll “provide continuous updates through the Support Portal”. Furthermore, they will “provide full transparency on how this occurred and provide steps being taken to prevent anything like this from happening again.”
While this could be a lesson on having excellent public relations, as these events can occur, it’s also a great reminder to consider the risk of changing an existing set-up or adding something new. In addition, if you have an outage on your primary systems, what back-ups do you have in place? What is your business continuity plan?
The FFIEC advises reviewing your Business Continuity Planning (BCP) at least annually, if not more frequently, to ensure that operations can continue during an event such as this or other disasters that could happen.
Any time you have a change to an existing system or process, or when implementing newer ones, a risk assessment should be performed to ensure that a variety of risks are considered that could impact your financial institution and may require additional controls to mitigate such risks. On the ACH side, for example, OCC Bulletin 2006-39 advises this for any new ACH activities that require risk assessments and enhanced systems/controls while also considering risks of growing volumes, new or evolving transaction types, new participants and more third-party involvement. When implementing any new systems or updating existing ones, have a plan for testing in an experimental environment and perform updates when the system is offline and not in production. Utilize formal change management requests to assist with planning, oversight, project management, testing and implementation of any installations or updates needed to your systems.
Let me provide an example. Before joining EPCOR, I oversaw and managed wire transfer operations and initially started submitting and receiving wires through the Fedline Advantage® web application. Our backup for this process was a redundant hot site with a secondary Fedline® router at that location. We also had procedures and employee listings in place for the offline wire transfer process with the Federal Reserve, in which institutions call in their wires and verify with securely delivered code sheets. Thus, our wire BCP was (1) primarily Fedline®, (2) a redundant hot site with a secondary Fedline® router and (3) an offline wire process. We also had various warm sites ready to use for employees, as well as virtual desktops.
However, thanks to mergers, we began to grow and the wire volume processed manually from Fedline® began to overwhelm us. So, we shopped around and sent various formal Request for Proposals (RFP) to wire vendors. That prompted demos of the new wire transfer, in which we involved various departments that could be impacted, such as IT, audit and the departments sending wires. Particularly with IT, they could discuss server, database and hardware requirements, along with setting up test sites. Collectively, we worked as a group to set up and launch this wire application as a successful project. In the meantime, we performed a risk assessment on our wire transfer processing to consider new risks that could evolve and what controls could mitigate such risks. We also updated our BCP plan to reflect what happens next in the event that our new wire application goes down. We retained Fedline Advantage® because we processed Fedwire® Securities and that acted as our backup. We retained the offline wires process because it was a backup to Fedwire® Securities.
One day, a database update was applied during production time and rendered our new wire software application useless. Unfortunately, it updated the database on the wire application on both production and the redundant site… Thus, we went back to Fedline Advantage® web application for the day to process wire transfers as a backup (a few settings on Fedline Advantage® had to be changed, along with contacting the Federal Reserve for temporary incoming wire arrangements). It was a tough day processing wires on a different system than we were used to, but it proved our BCP plan worked, and we were able to reasonably assume wire operations through Fedline Advantage® that day.
As an FYI on wire transfer BCPs, the Federal Reserve will retire the Fedwire® Offline service on December 31, 2024. That’s right, no more code sheets or calling in wires. On January 1, 2025, the offline service will be replaced by an import function on the FedPayments Manager service. If you have a BCP plan that may need updating, it’s a great time to reach out to your Federal Reserve account rep for any questions. For more information, check out these links:
BCP plan for wire transfers could consider the following tiers:
- Software or correspondent financial institution applications (+ redundant site)
- Fedline Advantage® application (+ redundant connection site)
- FedPayments® Manager import function (replaces offline after 12/31/24)
How many tiers should you go? That depends on the size, volume and complexity of your wire transfer operations. Also, if you’re engaging in any investment or trust client securities purchases or sales, the need for Fedwire Securities should also be considered for backup systems. For now, Fedwire Securities can only be processed on Fedline Advantage® and FedPayments® Manager could be the BCP plan for your security wire process.
Want some assistance with your BCP or anything else on your list? Our team of experts is happy to assist! Reach out to us at advisoryservices@epcor.org to learn how we can help.