[Resolved] [Service Disruption] Error with Advance Networking

Posted almost 5 years ago by Jake Yip

J
Jake Yip Admin

We are currently experiencing an issue with Advance Networking. Users might be seeing problems creating new tenant networks and other network resources. We are working on it and will post an update when this is fixed.

0 Votes


5 Comments

Sorted by
J

Jake Yip posted over 4 years ago Admin

All the virtual machines have been migrated away from this host.


This will be the last update for this disruption notice.


If you are still experiencing any disruptions, please feel free to open a support ticket.

0 Votes

J

Jake Yip posted over 4 years ago Admin

We have worked through most of the corruption and tests are stable now.


There is still ongoing work to migrate users' instances off the bad hypervisor, fixing it and rejoining it to the fleet.


We will post an update after this is done.

0 Votes

C

Chris Bayliss posted over 4 years ago

Thanks for the update

0 Votes

J

Jake Yip posted over 4 years ago Admin

We have a workaround for now, and working on a fix.


We have identified the cause to be a misbehaving hypervisor that is locking up the SDN database and timing out, leaving a stale lock and blocking subsequent updates for all other hypervisors. We have isolated this hypervisor for now.


You should now be able to create Networks and other Advance Networking resources without an error.


This has caused some corruption in our database. One such corruption is with Floating IPs - a few of them are in an inconsistent state. They are marked as 'free' in Neutron but are assigned in SDN database. When a user creates a Floating IP and is assigned one of these IPs by Neutron, this IP will not be usable. The workaround is to delete this IP and create another one.


We are working through the corruption today and will post an update shortly.

0 Votes

C

Chris Bayliss posted over 4 years ago

 Is this still ongoing? I'm trying to create a floating IP and keep getting an error.

0 Votes

Login or Sign up to post a comment