[SERVICE DISRUPTION] Unexpected Nectar-wide disruptions
As of the morning of Thursday 2018-12-13 we have become aware of widespread disruption to services across Nectar. In particular, at least the following problems have been noticed so far:
- Inability to access the Nectar dashboard due to authentication failures https://dashboard.rc.nectar.org.au
- Timeouts when using the OpenStack API clients
- Failures of various common functions such as creating/deleting instances and instance snapshots
We are still investigating the cause and will update this advisory as more information comes to light.
08:50 Update: One of the central hypervisors went down, and there seem to be problems handling requests to the remaining hypervisors. Further investigations underway.
10:30 Update: The problem was caused by a faulty network card. The services running on the affected hypervisor are being rebuilt/migrated elsewhere.