[Resoloved] [Unplanned outage] monash zone external network (Internet and Campus) Wednesday Jan 18, 2017 from 10:50am and ongoing

There was a urgent reboot of core gateway router at Monash as part of troubleshooting DC move related networking issues.


Date: 18 Jan 2017

Time: 10:50am

AZs affected: monash-01 and monash-02


Impact:

Internet access and inter-subnet routing for e.g.:

* NeCTAR monash-01 and monash-02 research cloud zones


Intra-subnet traffic, e.g., between cloud guests in the same zone and to volume storage services should not be impacted.



Update at 7:45pm 18/1/2017:

The eResearch network connectivity underpinning the monash-02 zone of the research cloud is now back online as of approximately 7pm this evening. Please accept our apologies for this major unplanned outage. If you require assistance recovering services please contact support@nectar.org.au.


Cause:

Root cause analysis is not yet complete but has been narrowed down to a switch redundancy error. As monash-02 is migrated to Monash's new datacentre over the coming month (see this forum notice for background) this network infrastructure is being retired and upgraded.


Update: 

monash-01 is back online at 11:00am 18 Jan 2017

Update:

We regret to advise that monash-02 remains inaccessible from outside the Monash eResearch Network. eResearch and eSolutions staff have determined that a router firmware update is causing physical link issues and are working to establish a workaround as soon as possible. We will post another update when an ETA or resolution is known. We sincerely apologise for any inconvenience this outage is causing!

Update:

The eResearch network connectivity underpinning the monash-02 zone of the research cloud is now back online as of approximately 7pm this evening. Please accept our apologies for this major unplanned outage. If you require assistance recovering services please contact support@nectar.org.au.


Cause:

Root cause analysis is not yet complete but has been narrowed down to a switch redundancy error. As monash-02 is migrated to Monash's new datacentre over the coming month (see this forum notice for background) this network infrastructure is being retired and upgraded.