[Hazard Notice] Nectar Cloud degraded during upgrade period

We are currently upgrading the Compute Service from Ocata to Pike and have noticed an increase in API errors (HTTP 500). Some of the requests that are returning errors include requesting for VNC console, attaching volumes, etc.

As a workaround, retrying the same requests should work.

We are looking for a solution and will update this notice at 1700 AEDT.

Update at 1700 AEDT: We are still working on this issue. Neutron (Network Service) is also impacted. We will provide an update when we have more information.

Update on 2018-11-23 1600 AEDT: We have rolled back a few services and have generally recovered most Compute Service and Network Service components. Some lingering issues remain, particularly around creating and deleting of security groups. We continue investigating these issues and will upgrade when we have more information.

Update on 2018-11-26 1230 AEDT: We have fixed the issue with Network Service. Checks are showing green at the moment. 

We are monitoring the situation and advise on when we plan to roll forward with the upgrade to Compute Service again.

Update on 2018-11-29 1500 AEDT

All checks are green at the moment.

We have narrowed down the bugs to certain libraries and have downgraded the affected libraries. We are continuing with the upgrade to Compute Service.

Login or Signup to post a comment