Dallas Network Outage (8/10/2018) (Investigating)
  • Priority - High
  • Affecting System - Carrier-1
  • At approximately 4:50pm CST, we experienced a catastrophic failure of our primary Juniper router at the Carrier-1 facility in Dallas, TX. It was immediately obvious that routing did not failover properly as designed so engineers power cycled the primary router and began identifying and correcting what seemed to be a buffer overflow that caused it to freeze. Software changes were made that should permanently correct the issue and also a line card was replaced just to be sure. Engineers are still testing and verifying at this time, but we do not expect additional downtime though there is a small risk for latency or packetloss in the coming hours. Juniper's emergency response team was also contacted within minutes to assist. The facility has the highest level contract with Juniper JTAC for support and hardware replacement.

    By approximately 5:37pm CST, the primary router was back online and all identified issues with redundancy were also resolved. However, we are now planning additional VRRP (Virtual Router Redundancy Protocol) conversions within our portion of the network at Carrier 1 in Dallas. This will be occurring in the upcoming week or two and again no additional downtime is expected. With the addition of VRRP in each of our cabinets, this type of issue cannot happen no matter what type of failure the primary router experiences (software, hardware, or human error). We will spare no expense and leave no stone unturned to ensure this incident and nothing like it can happen again. We greatly value your business and downtime is unacceptable to everyone. Rest assured that our efforts to PERMANENTLY resolve this will be successful. Please contact us if you have any further questions or concerns.

  • Date - 08/10/2018 14:50 - 08/10/2018 15:37
  • Last Updated - 08/10/2018 16:53

Powered by WHMCompleteSolution