technologywithin - Leased Line Outage – Incident details

All systems operational

Leased Line Outage

Resolved
Major outage
Started over 3 years agoLasted about 4 hours

Affected

Leased line connectivity

Major outage from 5:16 PM to 9:36 PM

Updates
  • Resolved
    UTC
    Resolved

    We are now seeing all lines back up.

    A full RFO will be requested from the carrier and will be issued when we have it.

  • Monitoring
    UTC
    Monitoring

    The carrier have moved the majority of the interfaces over to new hardware and are seeing circuits coming back up. There remain 2 interfaces which are causing issues which they are working to address.

    We will continue to monitor until all circuits effected by this outage are back up.

  • Update
    UTC
    Update

    The carrier has identified failed hardware and is in the process of replacing.

    They are also actively moving affected interfaces to alternative hardware.

    There is currently no ETA on resolution.

  • Update
    UTC
    Update

    The carrier has finished the remote reload of their equipment but this has not resolved the issues with their equipment.

    They have dispatched engineers to work on the issue at the data centre.

  • Identified
    UTC
    Identified

    The carrier is currently doing a complete emergency reload of their Telehouse West core.

    All leased lines that run through this equipment will be down until the rebuild has been completed.

    We currently don't have an ETA on this being completed.

  • Investigating
    UTC
    Investigating

    We have been alerted to an issue with one of our Leased Line Providers. They are investigating several thousand dropped leased line connections on their equipment located in Telehouse West.

    They are seeing some sites come back online but then go back offline again.

    We will update when we have more information.

    We are sorry for any disruption this may cause.