“Why are you in Backups?”
A question I got asked recently got me thinking. The backstory of how I got into the profession is a fairly boring one which includes an offsite event, some alcohol and a proposition from an old Manager of mine. Not very exciting.
The real story here is: “why have I stuck around as a Business Continuity expert?”
It started off as a challenge to automate the monotonous backup operations. Included the panic of receiving restore requests when you know a backup failure occurred around the same dates. All the way up to the sense of responsibility in knowing that your mistakes may cost the Company dearly for years to come.
As my career progressed, it really became about one thing though. The Superman complex.
It wasn’t for 3 years that I experienced my first DR event. As the name implies, having an IT disaster occur was completely unplanned. We had no forewarning and the outcome of the next few days was a testament to the planning, processes and procedures that were put in place many years previously.
In our shared Data Center, we had a plethora of visitors each day. Hosted Customers tending to their systems, new installs, upgrades, break fixes. One of these visitors was a Vendor technician who came through the access controlled doors to replace some SAN hard drives for a customer. His chaperon, having better things to do than watch the fix, told the technician to knock on the door when he’s done so that he may be let out.
As these things happen, the technician finished and decided to try and leave the Data Center. The longer he struggled to get out, the more panicked he became. And without realizing what he was doing, he hit the switch next to the door which looked like a big red exit button…
It was in fact, a trigger for the fire suppression system…
The system went off with massive clouds of smoke, noise and thunder. The technician allegedly passed out and his chaperon came rushing back when he heard the commotion. Suffice to say, while these fire suppression systems are great at suppressing fire, they are a fairly bad thing to have happen to a fully functioning Data Center. The vibrations caused by the release had damaged hundreds of hard drives. Systems were blue screening, data drives were dead and SAN’s were blinking with more red lights than can be found in Amsterdam.
Within minutes of realizing what had happened, we started receiving our first restore requests. Dozens of client servers all vying for a spot in the restore train.
After a solid two days of queued restores, prioritization and many, many re-assuring phone calls to clients, we finally completed the last of the immediate recoveries.
We managed a 100% restore success rate. Minimized downtime for our clients and we received more praise in those two days than the following two years.
In my youth I imagined that I was Superman, and the Flash. Acting out how I would swoop in and save the damsel in distress.
In reality, I do not come from Krypton and I didn’t get bitten by a radio active spider.
But as a Business Continuity expert, I get to experience all the highs of saving the business damsel in distress!