[Resolved] [Service Disruption] Error with Advance Networking
Posted
almost 5 years ago
by Jake Yip
J
Jake YipAdmin
We are currently experiencing an issue with Advance Networking. Users might be seeing problems creating new tenant networks and other network resources. We are working on it and will post an update when this is fixed.
0 Votes
5 Comments
Sorted by
J
Jake Yipposted
almost 5 years ago
Admin
All the virtual machines have been migrated away from this host.
This will be the last update for this disruption notice.
If you are still experiencing any disruptions, please feel free to open a support ticket.
0 Votes
J
Jake Yipposted
almost 5 years ago
Admin
We have worked through most of the corruption and tests are stable now.
There is still ongoing work to migrate users' instances off the bad hypervisor, fixing it and rejoining it to the fleet.
We will post an update after this is done.
0 Votes
C
Chris Baylissposted
almost 5 years ago
Thanks for the update
0 Votes
J
Jake Yipposted
almost 5 years ago
Admin
We have a workaround for now, and working on a fix.
We have identified the cause to be a misbehaving hypervisor that is locking up the SDN database and timing out, leaving a stale lock and blocking subsequent updates for all other hypervisors. We have isolated this hypervisor for now.
You should now be able to create Networks and other Advance Networking resources without an error.
This has caused some corruption in our database. One such corruption is with Floating IPs - a few of them are in an inconsistent state. They are marked as 'free' in Neutron but are assigned in SDN database. When a user creates a Floating IP and is assigned one of these IPs by Neutron, this IP will not be usable. The workaround is to delete this IP and create another one.
We are working through the corruption today and will post an update shortly.
0 Votes
C
Chris Baylissposted
almost 5 years ago
Is this still ongoing? I'm trying to create a floating IP and keep getting an error.
We are currently experiencing an issue with Advance Networking. Users might be seeing problems creating new tenant networks and other network resources. We are working on it and will post an update when this is fixed.
0 Votes
5 Comments
Jake Yip posted almost 5 years ago Admin
All the virtual machines have been migrated away from this host.
This will be the last update for this disruption notice.
If you are still experiencing any disruptions, please feel free to open a support ticket.
0 Votes
Jake Yip posted almost 5 years ago Admin
We have worked through most of the corruption and tests are stable now.
There is still ongoing work to migrate users' instances off the bad hypervisor, fixing it and rejoining it to the fleet.
We will post an update after this is done.
0 Votes
Chris Bayliss posted almost 5 years ago
Thanks for the update
0 Votes
Jake Yip posted almost 5 years ago Admin
We have a workaround for now, and working on a fix.
We have identified the cause to be a misbehaving hypervisor that is locking up the SDN database and timing out, leaving a stale lock and blocking subsequent updates for all other hypervisors. We have isolated this hypervisor for now.
You should now be able to create Networks and other Advance Networking resources without an error.
This has caused some corruption in our database. One such corruption is with Floating IPs - a few of them are in an inconsistent state. They are marked as 'free' in Neutron but are assigned in SDN database. When a user creates a Floating IP and is assigned one of these IPs by Neutron, this IP will not be usable. The workaround is to delete this IP and create another one.
We are working through the corruption today and will post an update shortly.
0 Votes
Chris Bayliss posted almost 5 years ago
Is this still ongoing? I'm trying to create a floating IP and keep getting an error.
0 Votes
Login or Sign up to post a comment