1. Handling Downtime: Job Well Done

    Yesterday, if you tried to call me (either at work, or at home, or even my mobile), you were probably unable to reach me.

    The telephone (VOIP) wholesaler I use, Unlimitel, was down for around 9 hours. Normally, 9h of downtime would really bother me, but I can honestly say I've never been happier with their service.

    Read on for some background info on why.Let me list a few points:

    • This is the first significant downtime we've had in 2 years (since I started using Unlimitel)
    • We pay 1.1¢/min for on-net service (local calls in a number of Canadian cities) and 2¢ for the rest of North America. Low rates for most of the rest of the world, too. $2.50/month for our DIDs. I don't expect completely flawless service for this price.
    • Stephan, the president, emailed customers to explain the problem. The first mail was before we even noticed that calls weren't working.
    • The actual cause of the accident was a screwup at Rogers (NOT Unlimitel's fault):
      • A truck accident somehow caused a bundle of fiber at Rogers to be cut. (Initial reports were that the lines were cut by a backhoe.)
      • Rogers' redundancy somehow failed. The actual cut was 25KM away from Unlimitel's datacenter.
    • Throughout the downtime, Stephan kept us well-informed of the situation, relaying ETA data from Rogers whenever possible.
    • Unlimitel routed all possible traffic to non-Rogers circuits as soon as possible. (Outbound calls started working, but inbound lines are on Rogers, and Rogers' redundancy failed, as mentioned.)
    • The few times that Unlimitel has actually made mistakes, they've kept us well-informed, and owned up to these mistakes quickly (they implemented CallerID poorly, a while back, and quickly fixed it, for example).

    All told, I'm very happy with them. I can understand why some people would be upset over ~9h of unplanned downtime, but all things considered, I think Unlimitel did an excellent job of handling the crisis.

    For what we pay, I couldn't expect better. Kudos to the team over at Unlmitel.