[Resolved] Tasmania and Tasmania-s Availability Zones
M
Matthew Armsby
started a topic
about 1 year ago
This incident has been resolved.
There has been an unplanned network disruption to the Tasmania and Tasmania-s Availability Zones between 4:30PM to 4:41PM AEST April 14, 2023.
As of 4:41PM AEST. April 14, 2023 the storage systems are recovering, which will impact the performance of instances in Tasmania and Tasmania-s Availability Zones.
We are currently monitoring the issue as a high priority and will provide updates.
Expected impacts:
* Very poor response of instances for disk reading/writing
* Instance creation/starting/stopping/deletion major delays
UPDATE 2023-04-14T20:00:00 AEST Performance is very poor, and is expected to remain very poor through the night. UPDATE 2023-04-15T09:30:00 AEST Performance is very poor, and is expected to remain very poor through the day. Recovery is slowly improving, no clear resolution, expecting a similar update in 24 hours. UPDATE 2023-04-17T10:30:00 AEST Performance is very poor, and is expected to remain very poor through the day. Recovery is slowly improving, with no clear resolution date. Storage for Tasmania and Tasmania-s availability zones should be considered unavailable.
UPDATE 2023-04-18T16:30:00 AEST Launches in Tasmania zone are possible once again, however the zone is not considered usable at this time.
UPDATE 2023-04-19T11:30:00 AEST Performance has improved, but is expected to remain inconsistent through the day. We have observed both normal behaviour, and a fluctuating unpredictable number of operations that are blocked for more than an hour. This is expected to result in unusual behaviour, and is not recommended to run workloads. Storage for Tasmania and Tasmania-s availability zones should be considered unavailable. UPDATE 2023-04-19T14:50:00 AEST The issue preventing launches in Tasmania-s has been resolved, however it remains affected by fluctuating storage performance. UPDATE 2023-04-20T10:10:00 AEST Storage recovery is ongoing. It is likely most instances will be functioning correctly but fluctuating storage performance will remain a problem.
UPDATE 2023-04-21T12:15:00 AEST Performance has improved, but is expected to remain inconsistent through the day. We have observed both normal behaviour, and a fluctuating unpredictable number of operations that are blocked for more than an hour. This is expected to result in unusual behaviour, and is not recommended to run workloads. Storage for Tasmania and Tasmania-s availability zones should be considered unavailable. UPDATE 2023-04-24T14:35:00 AEST Maintenance on the cluster is expected to produce some performance inconsistency through the day. We expect an unpredictable number of operations to be blocked for long periods of time. This is expected to result in unusual behaviour, and it not recommended to run workloads. Storage for Tasmania and Tasmania-a availability zones should be considered unavailable. UPDATE 2023-04-27T09:30:00 AEST Tasmania and Tasmania-s have returned to normal service. It is recommended instances are rebooted and file systems checked to resolve any unusual state occurring during this incident.
Matthew Armsby
This incident has been resolved.
There has been an unplanned network disruption to the Tasmania and Tasmania-s Availability Zones between 4:30PM to 4:41PM AEST April 14, 2023.
As of 4:41PM AEST. April 14, 2023 the storage systems are recovering, which will impact the performance of instances in Tasmania and Tasmania-s Availability Zones.
We are currently monitoring the issue as a high priority and will provide updates.
Expected impacts:
* Very poor response of instances for disk reading/writing
* Instance creation/starting/stopping/deletion major delays
UPDATE 2023-04-14T20:00:00 AEST Performance is very poor, and is expected to remain very poor through the night.
UPDATE 2023-04-15T09:30:00 AEST Performance is very poor, and is expected to remain very poor through the day. Recovery is slowly improving, no clear resolution, expecting a similar update in 24 hours.
UPDATE 2023-04-17T10:30:00 AEST Performance is very poor, and is expected to remain very poor through the day. Recovery is slowly improving, with no clear resolution date. Storage for Tasmania and Tasmania-s availability zones should be considered unavailable.
UPDATE 2023-04-18T16:30:00 AEST Launches in Tasmania zone are possible once again, however the zone is not considered usable at this time.
UPDATE 2023-04-19T10:30:00 AEST Outage notifications for 'Tasmania-02' zone have been moved to https://support.ehelp.edu.au/support/discussions/topics/6000067899
UPDATE 2023-04-19T11:30:00 AEST Performance has improved, but is expected to remain inconsistent through the day. We have observed both normal behaviour, and a fluctuating unpredictable number of operations that are blocked for more than an hour. This is expected to result in unusual behaviour, and is not recommended to run workloads. Storage for Tasmania and Tasmania-s availability zones should be considered unavailable.
UPDATE 2023-04-19T14:50:00 AEST The issue preventing launches in Tasmania-s has been resolved, however it remains affected by fluctuating storage performance.
UPDATE 2023-04-20T10:10:00 AEST Storage recovery is ongoing. It is likely most instances will be functioning correctly but fluctuating storage performance will remain a problem.
UPDATE 2023-04-21T12:15:00 AEST Performance has improved, but is expected to remain inconsistent through the day. We have observed both normal behaviour, and a fluctuating unpredictable number of operations that are blocked for more than an hour. This is expected to result in unusual behaviour, and is not recommended to run workloads. Storage for Tasmania and Tasmania-s availability zones should be considered unavailable.
UPDATE 2023-04-24T14:35:00 AEST
Maintenance on the cluster is expected to produce some performance inconsistency through the day. We expect an unpredictable number of operations to be blocked for long periods of time. This is expected to result in unusual behaviour, and it not recommended to run workloads. Storage for Tasmania and Tasmania-a availability zones should be considered unavailable.
UPDATE 2023-04-27T09:30:00 AEST Tasmania and Tasmania-s have returned to normal service. It is recommended instances are rebooted and file systems checked to resolve any unusual state occurring during this incident.