[SERVICE DISRUPTION] Unexpected Nectar-wide disruptions

As of the morning of Thursday 2018-12-13 we have become aware of widespread disruption to services across Nectar. In particular, at least the following problems have been noticed so far:

  • Inability to access the Nectar dashboard due to authentication failures https://dashboard.rc.nectar.org.au
  • Timeouts when using the OpenStack API clients
  • Failures of various common functions such as creating/deleting instances and instance snapshots

We are still investigating the cause and will update this advisory as more information comes to light.

08:50 Update: One of the central hypervisors went down, and there seem to be problems handling requests to the remaining hypervisors. Further investigations underway.

10:30 Update: The problem was caused by a faulty network card. The services running on the affected hypervisor are being rebuilt/migrated elsewhere.

10:50 Update: We believe this problem is now resolved.

If you require support for any problems caused by this disruption, please raise a support ticket through one of the following two methods: