[Resolved] [Service Disruption] Error with Advance Networking
We are currently experiencing an issue with Advance Networking. Users might be seeing problems creating new tenant networks and other network resources. We are working on it and will post an update when this is fixed.
Is this still ongoing? I'm trying to create a floating IP and keep getting an error.
We have a workaround for now, and working on a fix.
We have identified the cause to be a misbehaving hypervisor that is locking up the SDN database and timing out, leaving a stale lock and blocking subsequent updates for all other hypervisors. We have isolated this hypervisor for now.
You should now be able to create Networks and other Advance Networking resources without an error.
This has caused some corruption in our database. One such corruption is with Floating IPs - a few of them are in an inconsistent state. They are marked as 'free' in Neutron but are assigned in SDN database. When a user creates a Floating IP and is assigned one of these IPs by Neutron, this IP will not be usable. The workaround is to delete this IP and create another one.
We are working through the corruption today and will post an update shortly.
Thanks for the update
We have worked through most of the corruption and tests are stable now.
There is still ongoing work to migrate users' instances off the bad hypervisor, fixing it and rejoining it to the fleet.
We will post an update after this is done.
All the virtual machines have been migrated away from this host.
This will be the last update for this disruption notice.
If you are still experiencing any disruptions, please feel free to open a support ticket.