We have a workaround for now, and working on a fix.

We have identified the cause to be a misbehaving hypervisor that is locking up the SDN database and timing out, leaving a stale lock and blocking subsequent updates for all other hypervisors. We have isolated this hypervisor for now.

You should now be able to create Networks and other Advance Networking resources without an error.

This has caused some corruption in our database. One such corruption is with Floating IPs - a few of them are in an inconsistent state. They are marked as 'free' in Neutron but are assigned in SDN database. When a user creates a Floating IP and is assigned one of these IPs by Neutron, this IP will not be usable. The workaround is to delete this IP and create another one.

We are working through the corruption today and will post an update shortly.

We have worked through most of the corruption and tests are stable now.

There is still ongoing work to migrate users' instances off the bad hypervisor, fixing it and rejoining it to the fleet.

We will post an update after this is done.

All the virtual machines have been migrated away from this host.

This will be the last update for this disruption notice.

If you are still experiencing any disruptions, please feel free to open a support ticket.