[Resolved] DC outage affecting Nectar services

Nectar services, including all Tasmanian Availability Zones, have been restored.

This issue has been resolved. 

Due to an outage in one of the DCs hosting our TPAC Nectar node various services across the Nectar cloud are unavailable or degraded.

Outages:

* Tasmania-s zone

* Tasmania zone

Degraded:

* Nectar dashboard - all parts (resolved)

* Openstack API - all parts (resolved)

Currently degraded:

* Server instance listing for tas/tas-s

* Trove instance listing for tas/tas-s

* Volume instance listing for tas/tas-s

It is hoped essential services can be restored quickly but there is no timeline for restoration of research instances.

UPDATE 16/05/2022 10:51

Essential services have been restored enabling dashboard and other API functions.

Instances in Tasmania-02 should be manageable again, via dashboard or API.

There is no estimated time for return to service of research instances in Tasmania/Tasmania-s

Update 16/05/2022 13:44:

Service recovery continues for Tasmania and Tasmania-s.

Update 16/05/2022 16:00

Storage cluster has been restored to service.

Nectar dashboard will display instances again, but management of them (starting/stopping/...) is not currently possible.

Update 16/05/2022 17:30

The storage systems for the Tasmania and Tasmania-s Availability Zones are not ready for use yet, we expect to provide our next update at 17 May, 10AM AEST. The Tasmania-02 Availability Zone is working as normal and is ready to use.

Update 17/05/2022 10:00

RDSI (NFS) storage is still unavailable

Tasmania-s Availability Zone has been restored for active use

Recovery of Tasmania Availability Zone is ongoing.

The Tasmania-02 Availability Zone is working as normal and is ready to use.

Next update will be 14:00 today

Update 17/05/2022 14:35

RDSI (NFS) storage is still unavailable

Recovery of Tasmania Availability Zone is ongoing with an increasing number of instances being brought online.

The Tasmania-02 and Tasmania-s Availability Zones are working as normal and is ready to use.

Next update will be 17:00 today

Update 17/05/2022 17:00

All Tasmanian Nectar Availability Zones have been restored and instances should now be functioning as usual.

RDSI/NFS data storage continues to be unavailable and we will work with our vendor to return it to active service.

Update 18/05/2022 17:30

RDSI/NFS data storage has been restored to active service.

0 Votes