Dallas, TX issues - Psychz/Carrier 1 (Resolved)
  • Priority - Critical
  • Affecting Other - AS40676 Routing
  • At approximately 2:54am CST, we received alerts that the network in Dallas, TX was offline.    A Psychz core router had gone offline in the facility.   Network engineers immediately began investigating and were able to bring the router back online by 3:45am CST.    We continue to await the full details on the cause and ultimate resolution.   

    Rest assured that Tier.Net is sparing no expense to establish failover outside of Psychz's routing equipment in Dallas.   We understand the negative impacts that these repeated network events can have on your business.   Our efforts underway include establishing routing via our own ASN, an additional non-Psychz POP in Dallas, and migration away from LACNIC ranges that previously caused issues.   These efforts remain underway but we are making steady progress and hope to provide more updates very soon.   Your patience and understanding are much appreciated!   

  • Date - 04/28/2019 02:54 - 04/28/2019 03:45
  • Last Updated - 04/28/2019 01:54
Dallas, TX issues - Psychz/Carrier-1 3/18/201 (Resolved)
  • Priority - Critical
  • Affecting Other - Dallas, TX
  • Dear Valued Customer,

    At approximately  3/18/2019 7:30am CST, our connectivity in Dallas, TX was disrupted suddenly.   Initial assessment by the datacenter was that a widespread power outage/battery backup failure had crashed both core routers.   OOB (out of band) failures complicated the issue.  The issues were resolved after software fixes were implemented at approximately 8:45am CST.  The datacenter is working on a full report and detailed plan to eliminate the possibility of this failure ever repeating.    However, at Tier.Net, we understand the incredibly negative impact this recent network instability has had on our clients.   We share your frustration and sincerely apologize for the inconveniences you have experienced.   Although we have been assured that the Psychz/Carrier-1 facility issues have been resolved, we understand the many well justified concerns.   We are working on solutions to entirely bypass the causes behind each of these issues.   The sole common denominator is the routing equipment at the facility itself.   Not only Tier.Net, but many other providers at this same facility have been impacted.   As this is upstream from our equipment, we have to rely on a 3rd party, and in no uncertain terms, they have been failing us (and our clients) recently.   At Tier.Net, we understand the importance of 100% SLA-backed uptime.   We are and have been working on every possible solution to this recent instability.

    Actions we HAVE taken:

    1) Earlier this year, we implemented LACP routing redundancy direct to each cabinet in Dallas.  This means that two Juniper routers are directly utilized at all times.   It takes a failure of BOTH routers to cause loss of connectivity.   Unfortunately, though it caused a long period of stability, the recent issues prove that it was not enough to provide adequate protection against these types of failures. 

    2) Prior to #1, the datacenter also replaced core routers and all line cards entirely.   Unfortunately, this does not provide 100% protection against software or power failures, despite each of these systems being redundant.

    3) Due to new LACNIC policies, LACNIC IPs have been moved to our own routing.   Unfortunately, at this moment we still rely on the core routers at the facility.   This has been the source of each recent issue.

    Actions we ARE taking:

    1) Tier.Net has its own ASN and is moving towards completely replacing all local routing that is upstream of our equipment currently.   0% of the problems we have had lately have been a result of any direct failure by Tier.Net or any of Tier.Net's wholly owned equipment.   Tier.Net wholly owns all rack level networking equipment and hardware.   All failures have occurred upstream of this.  It is clear that we need to take control from endpoint to endpoint.    

    2) Tier.Net is establishing a new presence at a nearby Dallas facility and details will be provided shortly.   We will utilize this facility to add a POP at the Carrier 1 facility as well as give clients an alternative physical location in Dallas, with similar DDOS protection, for those who have lost confidence in the current facility.   We will provide free migration assistance for anyone who chooses to migrate.  We will also be able to utilize this facility as an additional POP to Carrier 1/Psychz once our ASN is established and routing is moved to our equipment.  To be clear, we will NOT be forcing any migration, but not only will provide the option, but also utilize it to add additional resilience to the existing Carrier 1 location.

    3) Those with LACNIC IPs are also encouraged to switch to new ARIN IPs we have announced.   We will gladly assign a 2nd set of IPs for you to use as long as you need to migrate from the LACNIC ranges at no additional charge.   The LACNIC ranges start with 181.xxx or 191.xxx.   Please contact us for more details on this.  

    Tier.Net is immediately working on the above items at the highest priority as well as participating in ongoing communications with the existing facility to ensure that this does not happen ever again.     We sincerely appreciate your understanding of the situation and promise to be in contact with you with updates again soon.

    Sincerely,
    Your Tier.Net Staff

  • Date - 03/18/2019 07:30 - 03/18/2019 08:45
  • Last Updated - 03/18/2019 13:18
Dallas Network Outage (8/10/2018) (Resolved)
  • Priority - High
  • Affecting System - Carrier-1
  • At approximately 4:50pm CST, we experienced a catastrophic failure of our primary Juniper router at the Carrier-1 facility in Dallas, TX. It was immediately obvious that routing did not failover properly as designed so engineers power cycled the primary router and began identifying and correcting what seemed to be a buffer overflow that caused it to freeze. Software changes were made that should permanently correct the issue and also a line card was replaced just to be sure. Engineers are still testing and verifying at this time, but we do not expect additional downtime though there is a small risk for latency or packetloss in the coming hours. Juniper's emergency response team was also contacted within minutes to assist. The facility has the highest level contract with Juniper JTAC for support and hardware replacement.

    By approximately 5:37pm CST, the primary router was back online and all identified issues with redundancy were also resolved. However, we are now planning additional VRRP (Virtual Router Redundancy Protocol) conversions within our portion of the network at Carrier 1 in Dallas. This will be occurring in the upcoming week or two and again no additional downtime is expected. With the addition of VRRP in each of our cabinets, this type of issue cannot happen no matter what type of failure the primary router experiences (software, hardware, or human error). We will spare no expense and leave no stone unturned to ensure this incident and nothing like it can happen again. We greatly value your business and downtime is unacceptable to everyone. Rest assured that our efforts to PERMANENTLY resolve this will be successful. Please contact us if you have any further questions or concerns.

  • Date - 08/10/2018 14:50 - 08/10/2018 17:37
  • Last Updated - 12/18/2018 16:38
Dallas, TX (US Central) Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - Dallas TX datacenter
  • Hello Psychz Networks - Dallas, TX (US Central) Customers,

    Planned Time 1AM PDT  - 6AM PDT October 20, 2018 Backup Planned Time 1AM PDT  - 6AM PDT October 21, 2018 Work Window - 5 Hours Outage Estimated Duration: 5-20 minutes

    Best practices will be used to avoid any downtime with proper reversion techniques.

    Reason - Maintenance on a core router ( MX480 ) as well as placing core services back to their full redundant mode.

    If you have any questions about this maintenance or want more information on how it will relate to your specific services with us, please feel free to contact us.

  • Date - 10/20/2018 01:00 - 10/24/2018 10:03
  • Last Updated - 10/19/2018 16:54
Emergency Maintenance: PDU Replacement (Resolved)
  • Priority - Critical
  • Affecting System - North Carolina Rack 105 PDU 2
  • Dear Valued Clients,

    During a routine walkthrough we noticed that PDU 2 of Rack 105 in our Lumos Datacenters (DC74), Charlotte NC facility appeared to be malfunctioning. We have scheduled an emergency maintenance beginning at 1/19/2017 at 6PM EST. At this time we will shut down servers (gracefully) that are sourced from a single power supply on PDU 2 beginning at 6PM EST. Dual power supply servers and servers on PDU 1 will not be effected. We will attempt to have all servers up as soon as possible, we estimate the downtime to be approximately 30 minutes or less.

    We sincerely apologize for the inconvenience, if you have any questions please do not hesitate to contact us.

  • Date - 01/19/2018 18:00 - 01/19/2018 16:16
  • Last Updated - 01/19/2018 13:15
Service Impacting Emergency Maintenance: 6PM (Resolved)
  • Priority - Medium
  • Affecting System - Dallas Rack BI04
  • Thank you for your continued business. This is a service impacting emergency maintenance notification for December 7th 2017 beginning at 6PM EST. Cisco has identified a bug in the switching platform we use that requires an immediate upgrade. The upgrade window is scheduled for 1.5 hours, however downtime should be far less then that. Rest assured that every precaution has been taken, onsite technicians will be standing by and spare switches are immediately available.

    Once again, thank you for your continued business, feel free to contact us if you have any questions.

  • Date - 12/07/2017 18:00 - 01/19/2018 13:15
  • Last Updated - 12/06/2017 15:48
Service Affecting Maintenance Sat, November 5 (Resolved)
  • Priority - Medium
  • Affecting System - Cascade Divide
  • UPDATE 7:34AM PST - All servers up.   If you are still experiencing any issues.   Please file a ticket.   Welcome to Bend, OR!

    UPDATE 4:32AM PST - 50% of servers are networked, cabled, and powered up.   We continue to work on the remainder.   All networking and routing has been tested and given the OK.  

    UPDATE 1:33AM PST - All equipment has arrived at the Bend facility.   Staff is unpacking and racking the equipment into our cabinets as planned.  Switches will be powered up first followed by servers. 

    UPDATE 10:03PM PST - IP routing has been tested and confirmed except for a few stragglers.   Equipment is still enroute and more updates will follow shortly.

    UPDATE 7:38PM PST - All servers have been cleanly shut down.   We are now safely transporting them to the Bend facility.   So far so good!   Please check back for updates.


    Dear Valued Clients,

    This notice is to inform you that we will be relocating equipment at our Cascade Divide Roseburg, OR location to the Cascade Divide Bend, OR facility overnight Saturday, November 5th at 6pm PST. This relocation is necessary due to our requirements for a larger and more redundant facility. The new facility features more transit providers, additional redundancy, more space, and larger capacity in general. This relocation will indeed cause a service interruption on the night of November 5th, but it should be as minimal as humanly possible. There will be NO IP space changes, rack-level networking changes, and the new facility is in the same geographic region and state. We chose the most off-peak time while allowing for unforeseen conditions to complete the maintenance with plenty of time to spare before the following morning. Your data, IPs, and configurations are NOT at risk. There will be no functional difference to your service, though you can look forward to enhanced redundancy and reliability, plus pricing benefits for bandwidth and much more in the future. Updates as we go will be posted at https://billing.tier.net/serverstatus.php 

    Details:

    If you have a dedicated or colo server, we suggest shutting it down (halting) it before November 5th at 6PM PST. If that is not possible, our staff will also begin powering down equipment at that time. We will attempt to shut down all servers cleanly in all cases (using CTRL+ALT+DEL where possible, or by logging in and halting). We will also cleanly shut down all VPS accounts and shared servers. Servers will all be booted up and checked once they arrive at the Bend, OR facility.

    The physical relocation will begin by 7pm PST. We have at least one staff member assisting PER CABINET so the equipment will be loaded and relocated as quickly as possible. We expect that equipment will be at the Bend facility and powering up within 4 hours. The maintenance period will extend to 3AM PST on November 6th to account for any unforeseen issues. Please check our network status page for updates and be aware that we will be busy with phonecalls and support requests the entire night. If possible, refrain from contacting us for status updates until we announce that servers are racked/cabled and should be powered back up.

    Your business is appreciated! Please contact us if you have any questions, concerns, or special instructions for us during this relocation. Our goal is to ensure clients waking up Sunday morning will simply return with ease to business as usual. We will post status updates as they become available on the night of November 5th. We will also post the Bend, OR datasheet and info to our site within the coming days. Thank you for choosing us!

  • Date - 11/05/2016 18:00
  • Last Updated - 11/06/2016 08:19
**EMERGENCY** Scheduled Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - Peer1 LA
  • Dear Customer,

    Please be advised that we will be performing scheduled network maintenance in our Los Angeles (West 7th Street) facility during the following date and time:

    From: July 22, 2016 - 00:00 PDT (July 22, 07:00 UTC)

    To:   July 22, 2016 - 02:00 PDT (July 22, 09:00 UTC)

    The window will occur on Friday July 22nd from 00:00 - 02:00. During this timeframe, Network Engineers will reboot a virtual switch chassis, due to the nature of the maintenance downtime of around 10 minutes is expected during this window and services will be affected.

    This work will be SERVICE IMPACTING. The appropriate staff will be present for the entire duration of this maintenance window.

  • Date - 07/22/2016 00:00 - 08/09/2016 11:12
  • Last Updated - 07/21/2016 18:56
Cascade Divide Datacenter Outage 7/18 (Resolved)
  • Priority - Critical
  • Affecting Other - Cascade Divide Datacenter
  • 07/18/2016 15:01 PST: We are currently investigating an outage in our Cascade Divide, Roseburg OR datacenter. Datacenter staff is aware of the outage. Updates to follow.

    7/18/2016 15:49 PST: We have been advised by the datacenter that their primary router has failed, and failover to the backup router failed. A manual failover is currently taking place.

  • Date - 07/18/2016 15:01 - 07/18/2016 20:38
  • Last Updated - 07/18/2016 15:52
Cascade Divide Datacenter Outage (Resolved)
  • Priority - Critical
  • Affecting System - Cascade Divide Datacenter
  • At 10:57 AM PST (GMT-8) our NOC team noticed an outage at our Cascade Divide, Roseburg OR datacenter. The NOC made contact with the datacenter and they are aware of the issue, and investigating.

    UPDATE: 11:12 AM PST -- There is a confirmed fiber cut which is affecting all circuits (including redundant and protected circuits) in and out of the datacenter.

    UPDATE: 11:23 AM PST -- Response teams are en route from all major fiber providers to the facility (LSN, HE, Level3, etc). We do not yet have any ETA.

    UPDATE: 12:04 PM PST -- Teams are on site from all carriers and repairs are underway. Substantial damage has been done to the multiple fiber paths and utility poles in the area, including full fiber sheath cuts. Crews are giving a rough estimate of 4-5 hours.

    UPDATE: 1:55PM PST -- Crews have updated their original estimate. The new expected resolution time is 7:00PM PST. Our on-scene technicians have shared pictures, available at: http://imgur.com/a/023V2

    UPDATE: 4:06PM PST -- Aerial lines have been pulled across roadways, and work is underway on terminating and splicing the fiber bundles. The ETA remains the same.

    UPDATE: 5:47PM PST -- Service has been restored to the first fiber bundle. At this time if you have any remaining issues please open a ticket so we can address them.

  • Date - 04/13/2016 10:57 - 04/18/2016 10:16
  • Last Updated - 04/13/2016 17:49
Cascade Divide Network Issues (Resolved)
  • Priority - Critical
  • Affecting Other - Cascade Divide - All services in Roseburg, OR
  • 5/31/15 @ 7:30pm PST - there is a large DDOS in progress affecting upstream providers at our Oregon location.   We are working on this and should have it resolved momentarily.   This affects beta.tier.net shared srevers, VPS, and dedicated/colo in Roseburg, OR / Cascade Divide.

  • Date - 05/31/2015 19:39 - 05/01/0015 00:00
  • Last Updated - 05/31/2015 20:21
DC74 Datacenter Maintenance April 11th (Resolved)
  • Priority - Medium
  • Affecting Other - DC74 Datacenter
  • Thank you for your continued business!

    Please be advised of the following service impacting maintenance notification.

    Reason for Notification: Router Maintenance
    Location: DC74 Data Centers – CLT4 facility, 1612 Cross Beam Drive, Charlotte NC
    Start Date & Time: Saturday, April 11th, 2015 starting at 2300 EST


    Expected End Date & Time: Saturday, April 11th, 2015 ending at 2330 EST

    PLEASE NOTE: This affects DC74 datacenter ONLY.

    Description: We will be performing router maintenance potentially requiring a reboot of the routers. This will impact BGP routing and could potentially cause up to a 15-30 minute loss of network connectivity. The date and time chosen is the lowest traffic routing period for the entirety of the datacenter. Please be aware of this maintenance window and potential service disruption.

    Please let me us know if you have any questions.

  • Date - 04/11/2015 23:00 - 07/21/2016 18:55
  • Last Updated - 04/09/2015 18:29
12/3/14 Cascade Divide Final Network Upgrade (Resolved)
  • Priority - Medium
  • Affecting Other - Cascade Divide datacenter
  • Please note that we will be performing the final upgrade to our networking infrastructure at our Cascade Divide, Roseburg OR location starting December 3rd, 2014 at 11:30PM PST (GMT-8) and lasting until 12:00AM.

    You may experience a service disruption of approximately five minutes or less during this window.

    Thank you for your continued business.

  • Date - 12/03/2014 23:30 - 07/21/2016 18:55
  • Last Updated - 11/30/2014 20:37
[SERVICE IMPACTING MAINTENANCE] Sat. 11/29/20 (Resolved)
  • Priority - Medium
  • Affecting System - DC74 Datacenter
  • Thank you for your continued business! This is a service impacting maintenance notification scheduled for Saturday, 11/29/2014 beginning at 10PM EST (GMT-5) for our DC74, Charlotte NC facility.

    At this time we will be upgrading our core switching infrastructure. Clients will be going down one by one as you are moved to the new switch stack. We expect the maintenance window to be complete by 11PM EST (GMT-5). There is an expected 5-10 minutes of downtime per machine.

    If you have any questions, please do not hesitate to contact us.

  • Date - 11/29/2014 22:00 - 11/30/2014 20:33
  • Last Updated - 11/24/2014 15:54
SERVICE IMPACTING Maintenance notification fo (Resolved)
  • Priority - Medium
  • Affecting System - Cascade Divide, Roseburg OR
  • Thank you for your continued business! This is a SERVICE IMPACTING maintenance notification. This notification is for our Cascade Divide, Roseburg OR location.

    We will be performing switch upgrades in order to provide a more robust and redundant switching infrastructure.

    This upgrade is schedule to take place on: 9/6/2014 from 23:45 (11:45 PM) until 9/7/2014 00:15 (12:15 AM) GMT-8 (PST)

    Should you have any questions please do not hesitate to contact us regarding this issue.

  • Date - 09/06/2014 23:45 - 11/24/2014 15:09
  • Last Updated - 08/22/2014 17:50

Powered by WHMCompleteSolution