Who, me? It’s Monday, and this week’s column contains another reminder to check that these backups have actually worked on an unfortunately synchronized episode of Who, Me?
Our story comes from a reader we’ll call “John” (because it’s not his name) and takes place in the glory days of Windows NT 4 and Microsoft Exchange Server 5.5.
John had earned his prestige in the technical support department of a mail order parts company (remember those days before e-commerce?) And was promoted to the dizzying level of “the TI “.
“That was made up of me and another guy,” he noted.
Still, it meant John had to do all sorts of exciting work, such as processing data for catalog submissions to SQL Server. When this colleague (who was in charge of the domain and dealing with Exchange’s weaknesses) left, John handed him the keys to the server room. He even went on to take some courses to get an MCSE certification, thanks to a boss who considered himself a bit of a technician.
Time passed and nothing went wrong. The lights were kept on. The computers were orange. Clearly, John was doing a splendid job.
Until, he said, “the first domino fell.”
“The Exchange server,” John said, “that doubled as a fax gateway (because email wasn’t a thing yet and almost everyone received their account statement by fax; this weekly fax job was literally critical to the company’s cash flow) .the blue projection began. “
A look at the code of the Blue Screen of Death (BSOD) hinted that the motherboard might be to blame.
“The boss asks me if I’m ready for the task of rebuilding the Exchange,” John recalled. “I tactfully suggest that we help each other out: ‘I know a guy who’s great with Exchange.'”
Okay! The plan was to do the work Friday night. Everything is simple enough: back up your Exchange server to tape. Make a backup, just in case. Then write the hardware.
Everything went fine. The new motherboard came in. “We took the amazed 8-way fax card out of its grotesqueness and glued it to the new motherboard.”
By midnight Windows was back. The exchange is back. Everything was full of services. Data recovery tape only …
… That worked well. A few obscure orders from ESEUTIL were needed and … The exchange did not begin. Strange. The other backup has been tested. Again, Exchange did not start and talked about a corrupt database.
By now it was 2 in the morning and John decided to call it night. After all, I had the whole weekend left to get things up and running.
It’s not hard to guess what happened next. It turned out that ESEUTIL had enough options because the potential for error was immense. “Nothing worked,” John said, “no one had a mailbox anymore.”
That was bad. But not a disaster. Backups were run every day (one full on Sunday and then daily incremental), so restore them, right? They were restored … fingers crossed … Exchange resumed …
No luck. Repairs attempted with ESEUTIL took hours, but nothing worked.
At this point it was 4 in the morning on Sunday and the tools had to be lowered for a few hours to sleep.
“Tomorrow is coming,” John said, “and this time we’re trying to get the machine back with the old motherboard because we’re running out of ideas. Screw it off. Four hours later the same result. Let’s go back to the new motherboard because we’ve got it.” found more. ESEUTIL settings. Unfortunately. It turns out that all 36 backup tapes were corrupt. “
A weekend to sort things out had become just seven hours before the staff got to work.
It was Sunday at midnight, despair began.
“What’s better than not having any email service? Empty mailboxes!”
And so there was one last effort. Exchange has been reinstalled and configured from scratch. The directory has been restored. Sure everyone would go to work to find their mailbox empty, but at least it would be possible to send and receive messages from the outside world.
At 6 a.m., the duo stood at the front of the building, ready to dispense the bad news to the staff when they arrived.
“One by one the message is delivered” your email works but you don’t have any of your old emails “.
The responses were mixed: “The best of them was the store manager: ‘Don’t worry mate, it was all useless shit.'”
John’s head finally appeared (what was considered a technical genius). He followed with half an hour of questions and explanations before accepting John’s apologies: “Well, okay, you did everything you could. It’s okay. I have a backup of all my contacts about this …”
He picked up his new luxury Palm PDA, one that had dispensed with the cable for a wireless link.
“He walked into the office and prepared to retrieve his entire mailbox,” John recalled.
“Unless your Palm software sync settings said ‘take the latest'”.
“What’s newer than Friday’s full mailbox? Today’s void”
And with that the duo beat a hasty retreat. Much like the case of “I’ll take my coat …”
Have you ever received a slap in sync settings? Or have you seen a Friday night just a job explode on a catastrophic weekend? Of course you do, and you should share your story with an email to Who, Me? ®