[Resolved] [Hazard Notice] Intermittent database issues affecting APIs
We are experiencing intermittent database deadlocks with our databases powering the API servers. Most API calls are unaffected, as the servers can retry a failed database connection.
However, we are seeing some cases that have user impact.
- Magnum clusters might fail to spin up. Deleting the cluster and retrying works
- Some loadbalancers member might error out. We are cleaning them up and restoring them to a healthy state.
We are focused on fixing this and will provide more information when we can.