When Things Go Wrong: The Importance of a Backup and Recovery Plan

We recently got an alert that a client’s main router had gone offline, but their network was running fine the whole time. Below is what was really going on and how we fixed it, and then why I think recovery planning covers a lot more than file backups.

What Happened

The alert came from our monitoring and said the client’s primary network router had gone offline. That’s not an alert anyone likes to see. However, when we looked at the site, everything was still running. Traffic was flowing, employees could work, and internet access never dropped.

So if the network was fine, what was actually broken? The problem was in the router’s management layer. That’s the software IT staff use to configure, monitor and maintain the device, and it had stopped responding. The network itself was healthy, we just couldn’t manage it or make any changes to it.

The network management application had recently installed an automatic update. We worked through the standard troubleshooting steps and none of them fixed it, and there was no specific log entry or error pointing at the cause. So I decided to roll back the update, putting the application back on the previous version we knew was good. Full management access came right back. The underlying infrastructure had never failed at all, it only looked that way.

I think the lesson here is that a system that looks broken isn’t always broken in the way you think, so you diagnose before you act. If the plan had been to treat the router as dead and reset it, a network that was working fine would have ended up with a real outage.

Recovery Is More Than File Backups

When most people hear “backup and recovery” they think of files, making sure important documents aren’t lost if a hard drive dies. That absolutely matters. But recovery planning goes further than that.

Recovery planning basically means being able to put systems, software and configurations back to a known working state when something goes wrong. It means being able to answer one question: if this breaks today, how do we get back to working?

For network equipment, that means keeping track of which software and firmware versions are known to be stable, having configuration backups so a device can be restored or replaced quickly, knowing which version to roll back to if an update causes problems, and having the recovery steps written down so they’re clear and can be repeated. Without those things, this incident could have taken a lot longer to sort out, or ended with that full device reset and an outage nobody needed.

What Happens Without a Plan

If nothing is written down, troubleshooting turns into improvising, and improvising is slow. Someone has to figure out what version a device was running before the last update, go hunting for configuration exports that may not exist, or wait on vendor support for answers that should already be on hand.

That time costs something even when the business keeps running. Even a few hours without good management visibility is a risk. If something else had gone wrong on the network during that window, the ability to respond quickly would have been compromised. A recovery plan makes that window a lot shorter.

What I Think a Good Recovery Plan Includes

Every business is different, so there isn’t one template for this. These are the pieces I look for.

Know What You Have

You can’t recover what you don’t know exists. Start with an accurate inventory of the hardware, software versions, firmware and configurations for the systems your business depends on. It doesn’t have to be complicated. A maintained record of “what version is running on what device” is often enough to make recovery a lot faster when something breaks.

Keep Configuration Backups

For network equipment, servers and other infrastructure, I believe configuration backups are just as important as data backups. If a device fails or needs to be replaced, a current configuration export means restoring it takes minutes instead of hours. Many network management platforms can back up configurations automatically on a schedule. If yours doesn’t, that’s worth fixing.

Test That You Can Actually Recover

How do you know a backup works if you’ve never restored from it? You don’t. A backup that has never been tested is one you can’t fully trust, and recovery steps nobody has practiced take longer and add risk when you’re doing them under pressure. Checking now and then that backups are complete and that the steps actually work is pretty basic, and it’s also one of the things that gets skipped the most.

Write the Steps Down

The person fixing something isn’t always the person who set it up. Written runbooks, even simple ones, keep recovery steps consistent and keep the know-how from living only in one person’s head (and that person is always on vacation the day it breaks).

Keep a Way to Roll Back

Software updates are one of the most common reasons an otherwise stable system starts acting strange. Vendors release updates with good intentions, but bugs and compatibility issues happen. That’s exactly what happened here, with an automatic update, and rolling back is what got us out of trouble.

Wherever you can, keep the ability to roll software or firmware back to a previous version. That might mean keeping copies of older installers, learning how your platform’s rollback feature works, or trying updates in a test environment before pushing them to production systems.

Where to Start

This one ended well. Business never stopped, and the management problem was found and fixed quickly. I don’t think that was luck. We had the information and the rollback option ready when we needed them.

Recovery planning isn’t only for big companies with big IT budgets. Small and mid-sized businesses depend on their technology just as much, and without the resources to absorb it, a long stretch of downtime can hurt them even more. If you don’t have recovery steps written down, configuration backups, or a plan for when an important system fails, I’d start with the inventory and build from there.

Reboot Technology helps businesses in the greater Milwaukee area with exactly this. Whether you’re starting from nothing or tightening up what you already have, reach out and we’ll help you put together a recovery plan that fits your setup.