For those of you who work in IT, you know the feeling when you get “that” call. It’s the call that begins without the usual common conversational courtesies and quickly spells out a major change in the planned trajectory for the day. I received just such a call from a large client of ours last Tuesday. The CIO told me that, after changing a hard drive in their storage array, all of the servers hosted on that array were malfunctioning or reporting that they were down. For this customer, the only system unaffected was the system that takes and manages order (thank goodness). While being able to continue to process orders, services such as email, voice product picking, printing, time clocks, and more were unavailable. These are all essential components of their operation thus requiring expedient restoration of all services.
Over my career in IT, these calls happen several times per year and can cause varying levels of panic. The panic level directly depends on the customer in question and whether or not they had properly invested in the tools necessary to make an event like this a speed bump instead of a brick wall.
Luckily for (both of) us, this client had both prepared and properly invested. We had talked through this exact scenario and had enacted plans to address it. Thus, when the call came in, we immediately put our plans into place and began restoring their environment using the hardware that had specifically been set aside for this purpose. While those restores were running, we activated their emergency inbox service (the largest server that was impacted was their email server). The emergency inbox allowed all users to see the emails that were coming into their email address, as well as the capability to send outbound messages.
Within a few hours, the bulk of their systems were back online and operational. By that evening, everything was fully restored with zero unexpected issues. Had this client not worked with us to plan for this type of event and invest in solutions to mitigate it, the damage it could have caused to their business could have been extensive. Instead, they were fully functional the following morning and we shifted our focus back to planning and investment in technology solutions to further decrease the restore time should this ever happen again.
I share this story to remind all of you, both on the client side, as well as the technology side: plan for the worst case scenario. Understand the amount of downtime your customers can sustain and be sure that the solutions that are put into place will meet or exceed those requirements. In this time where health, business, and economic uncertainty is all around us, we must do everything that we can to mitigate risk and ensure that technology remains the tool that it must be to keeping business running.