Unscheduled NSW Incident - WAN
Incident Report for Hosted Network
Resolved
After monitoring for several days this issue now appears to be stable and resolved.

Engineers have determined that this issue was caused by a software issue within the Cisco device that was handling PPPoE termination for NSW which was causing internal BGP to drop which in turn interrupted network traffic between the device and the rest of the network.

Our engineering team are currently investigating additional configurations to add more redundancy and alternative solutions to minimise the service disruption should something like this occur again.

For further technical details on this outage please contact us at noc@hostednetwork.com.au
Posted 21 days ago. Sep 23, 2019 - 10:01 AEST
Monitoring
All services have been restored and systems appear to now be stable.

Our engineers are closely monitoring the systems at fault while investigating the root cause of the issue.

If you are still experiencing issues please start by rebooting the NTD and your router as your service may need this in order for force authentication to retry. If you still have issues please log a support ticket by emailing support@hostednetwork.com.au
Posted 25 days ago. Sep 19, 2019 - 17:33 AEST
Update
Unfortunately the work around put in place earlier today has failed and caused a major outage.

Engineers are working to rectify the issue and are working on replacing the failed piece of equipment.

Updates to follow as they are available.
Posted 25 days ago. Sep 19, 2019 - 15:33 AEST
Update
All NSW WAN services are now restored. However engineers are investigating the root cause of the failure.

If you are still experiencing issues please reboot your NTD and CPE and if issues persist please raise a support ticket by e-mailing support@hostednetwork.com.au
Posted 25 days ago. Sep 19, 2019 - 11:30 AEST
Identified
We have identified the issue as a failed piece of key hardware and are implementing workarounds at the moment.

- Services are starting to restore as we implement the workaround.
Posted 25 days ago. Sep 19, 2019 - 11:20 AEST
Investigating
Please be advised that we are currently experiencing an unscheduled network incident for NSW based WAN services and engineers are investigating both the impact and the root cause.

Updates will be provided as they become available.

If you have any questions regarding this incident please contact the network operations team at noc@hostednetwork.com.au or on 1300 781 148.
Posted 25 days ago. Sep 19, 2019 - 11:02 AEST
This incident affected: Hosted Desktop Services, Hosted Infrastructure Services, Hosted Storage Services, Cloud Backup Services, Voice over IP Services, International Network Services, Internet Services, LTE Services, Fixed Point Wireless, Web Hosting Services, and Colocation Services.