When you are responsible for IT systems Widespread Outages are part of the job.

Often these Outages are not even your fault. So many things go wrong with IT systems. For example, internet service providers have problems and there is nothing you can do there. Or power might go out to a building and there is nothing you can do there. Yes, you can have a UPS but you can only prepare for a short duration since most companies don’t want to invest in infrastructure.

This is not to be defensive or make excuses. This is just to say that when there are IT problems, it doesn’t indicate that your IT staff is incompetent. IT often have redundant systems, but there are problems with complexity as well. For example, most companies have two internet service providers but when one has a problem it can take up to 30 seconds for systems to switch over. Very few companies have invested to make internet switchover a seamless process.

For example I worked for a company that had frequent internet problems. When they switched to their backup system, it could take 30 seconds for all the systems to switch over. Part of the problem was the legacy software they were using. Another part was their overly complex networking structure. Way too many small networks which is typical of many businesses. It takes a great deal of effort to create an IT system that is as redundant as possible.

Too often when I have asked management to create and buy things to make things more reliable I have been told no. Then something happens and there is a big problem. I remind them that I asked to buy something cheaper that would have prevented this. The problem is always 10 times the cost of the solution. Too often, companies need to experience pain before they realize they should have prepared.

Which is fine. If a company wants to experience pain before they spend money that is their choice. I just find it interesting.

The post Dealing with widespread outages appeared first on Stuff worth knowing about.