[Backport][bug/1352826] applying iptables rules takes too long when large scale deployment
Bug #1399168 reported by on 2014-12-04
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Critical | | |||
| | Critical | | |||
| | Critical | | |||
| | Critical | | |||
| | Critical | |
Bug Description
This is a backport of
Original description
================I found the time to finishing the applying iptables rules( in neutron/agent/linux/iptables_manager.py _apply_synchronized,_modify_rules) takes nearly more than half an hour( 36 minutes in my environment) when the number of active vms in cloud is more than 980.
This will lead that the time of bringing new created port up when booting an instance will take very long, and if the vif_plugging_is_fatal is true, the vif_plugging_timeout is not big enough, booting will fail. Although optimization on _modify_rules in patch did help shorten the cost, but still the time is not short enough (it takes 17 minutes when the number of active vms in cloud is more than 980 in my environment). Further optimazation on _modify_rules need be done to fit the situation of Large-scale deployment. Tags:
on 2014-12-04
| Changed in mos: | |
| status: | Triaged → In Progress |
| wrote on 2014-12-04: |
| wrote on 2014-12-04: |
(3.3 KiB)
DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira12+git.f90d4c0.a299407Changeset:
project: openstack/neutron branch: openstack-ci/fuel-6.0/2014.2 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: patchset-createdFiles placed on repository:
neutron-common_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-dhcp-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-l3-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-lbaas-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-metadata-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-metering-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-bigswitch-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-bigswitch_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-brocade_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-cisco_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-hyperv_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-ibm-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-ibm_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-linuxbridge-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-linuxbridge_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-metaplugin_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-metering-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-midonet_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-ml2_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-mlnx-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-mlnx_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-nec-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-nec_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-nicira_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-oneconvergence-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-oneconvergence_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-openflow-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-openvswitch-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-openvswitch_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-plumgrid_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-ryu-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-ryu_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-vmware_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-plugin-vpn-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-server_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb neutron-vpn-agent_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.deb python-neutron_2014.2-fuel6.0~mira12+git.f90d4c0.a299407_all.debNOTE: Changeset is not merg...
on 2014-12-04
| tags: | added: scale |
| Changed in mos: | |
| importance: | High → Critical |
| wrote on 2014-12-05: |
Hm, did you make the same loading as for 5.1.1, so 6.0 - more than 980 vms? Let me remind you that we certified 5.1.1 only on 20 nodes. Aleksander, please clarify this moment for 5.1.1 version.
| wrote on 2014-12-06: |
Nastya, this is original description from the bug filed in upstream.
The RCA of this bug is algorithm for modifying iptables rules on each compute. So the main factor of this issue is a number of VMs per compute and number of security groups rules.
If we read comments from upstream issue we can find that bug submitter talks about 100 VMs per compute approximately:
"We have got 11 hosts, one is the controller, other 10 are compute nodes." (c)So I did several checks to verify the patch:
TestCase1 (measuring a time of updating iptables rules):
Prerequisites:
- ~500 rules in security group "test-sg" to attach for VMs - already deployed 96 VMs on the same compute using an availability-zone which contains the same compute for each VM - flavor is not important here, cirros image1. When all pre-deployed VMs are Active check that openvswich-agent doesn't consume 100% CPU
2. Start booting 1 new VM with "test-sg" security-group and start measuring time when ovs-agent consumes 100% CPUExpected result: 100% CPU usage for ~10 sec, VM is booted success
Note: before this fix you could check that VM is being booted too long (8-10 mins and could run into Error state) and CPU about 100% for the whole period of bootingSecond check is simple:
1. Deploy 10 VMs on dedicated compute, attach "test-sg" for each instance.
2. Deploy 20 VMs on dedicated compute, attach "test-sg" for each instance. 3 Deploy 50 VMs on dedicated compute, attach "test-sg" for each instance.Expected result: all 80 VMs are up and runnig
Note: before fix number of failed VMs was about 90%-30%All my tests are passed I assume that this bug is fixed correctly and proposing to merge it.
| wrote on 2014-12-06: |
Also I've performed the following check using WARNING logs in python at the beginning and ending of function which modifies security group rules using "START MODIFY RULES" and "END MODIFY RULES" labels:
After patch:
There was booted 96 VMs and 460 rules per VM
Adding +1 VM
2014-12-05 22:51:46.616 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] START MODIFY RULES 2014-12-05 22:51:57.647 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] END MODIFY RULESResult: 11 sec
Adding +1 VM
2014-12-05 22:55:35.525 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] START MODIFY RULES 2014-12-05 22:55:46.934 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] END MODIFY RULES Result: 11 secAfter deleting 4VMs
2014-12-05 23:01:21.946 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] START MODIFY RULES 2014-12-05 23:01:33.097 22962 WARNING neutron.agent.linux.iptables_manager [req-3029b20d-abef-4528-92ec-ba00d40afdb3 None] END MODIFY RULES Result: 11 secBefore patch
There was booted 96 VMs and 460 rules per VM
2014-12-05 23:28:30.195 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] START MODIFY RULES
2014-12-05 23:38:38.574 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] END MODIFY RULES Result: 10 minsBooted 1 VM and
2014-12-05 23:47:12.682 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] START MODIFY RULES 2014-12-05 23:57:13.310 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] END MODIFY RULESResult: operation of modifying iptables took 10 min and VM moved into Error state
Deleting of 20 VMs
2014-12-06 00:00:18.216 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] START MODIFY RULES 2014-12-06 00:10:04.423 30024 WARNING neutron.agent.linux.iptables_manager [req-b6968b33-ada2-4b51-a6ef-03527bc25415 None] END MODIFY RULESResult: operation of modifying iptables took 10 min
| wrote on 2014-12-06: |
I think patch is ready to be merged
| wrote on 2014-12-06: |
RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira13Changeset:
project: openstack/neutron branch: openstack-ci/fuel-6.0/2014.2 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: change-mergedFiles placed on repository:
openstack-neutron-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-bigswitch-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-brocade-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-cisco-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-hyperv-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-ibm-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-linuxbridge-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-mellanox-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-metaplugin-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-metering-agent-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-midonet-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-ml2-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-nec-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-nuage-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-ofagent-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-oneconvergence-nvsd-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-opencontrail-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-openvswitch-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-plumgrid-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-ryu-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-vmware-2014.2-fuel6.0.mira13.noarch.rpm openstack-neutron-vpn-agent-2014.2-fuel6.0.mira13.noarch.rpm python-neutron-2014.2-fuel6.0.mira13.noarch.rpmChangeset merged. Package placed on primary repository
RPM repository URL:| wrote on 2014-12-06: |
DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira12Changeset:
project: openstack/neutron branch: openstack-ci/fuel-6.0/2014.2 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: change-mergedFiles placed on repository:
neutron-common_2014.2-fuel6.0~mira12_all.deb neutron-dhcp-agent_2014.2-fuel6.0~mira12_all.deb neutron-l3-agent_2014.2-fuel6.0~mira12_all.deb neutron-lbaas-agent_2014.2-fuel6.0~mira12_all.deb neutron-metadata-agent_2014.2-fuel6.0~mira12_all.deb neutron-metering-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-bigswitch-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-bigswitch_2014.2-fuel6.0~mira12_all.deb neutron-plugin-brocade_2014.2-fuel6.0~mira12_all.deb neutron-plugin-cisco_2014.2-fuel6.0~mira12_all.deb neutron-plugin-hyperv_2014.2-fuel6.0~mira12_all.deb neutron-plugin-ibm-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-ibm_2014.2-fuel6.0~mira12_all.deb neutron-plugin-linuxbridge-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-linuxbridge_2014.2-fuel6.0~mira12_all.deb neutron-plugin-metaplugin_2014.2-fuel6.0~mira12_all.deb neutron-plugin-metering-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-midonet_2014.2-fuel6.0~mira12_all.deb neutron-plugin-ml2_2014.2-fuel6.0~mira12_all.deb neutron-plugin-mlnx-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-mlnx_2014.2-fuel6.0~mira12_all.deb neutron-plugin-nec-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-nec_2014.2-fuel6.0~mira12_all.deb neutron-plugin-nicira_2014.2-fuel6.0~mira12_all.deb neutron-plugin-oneconvergence-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-oneconvergence_2014.2-fuel6.0~mira12_all.deb neutron-plugin-openflow-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-openvswitch-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-openvswitch_2014.2-fuel6.0~mira12_all.deb neutron-plugin-plumgrid_2014.2-fuel6.0~mira12_all.deb neutron-plugin-ryu-agent_2014.2-fuel6.0~mira12_all.deb neutron-plugin-ryu_2014.2-fuel6.0~mira12_all.deb neutron-plugin-vmware_2014.2-fuel6.0~mira12_all.deb neutron-plugin-vpn-agent_2014.2-fuel6.0~mira12_all.deb neutron-server_2014.2-fuel6.0~mira12_all.deb neutron-vpn-agent_2014.2-fuel6.0~mira12_all.deb python-neutron_2014.2-fuel6.0~mira12_all.debChangeset merged. Package placed on primary repository
DEB repository URL:| wrote on 2014-12-07: |
Colleagues, why is it Incomplete for 5.1.2? I thought it affects 5.1.X. Let's figure out: if it does not, then the status of this bug should be Invalid for 5.1.1 / 5.1.2.
| wrote on 2014-12-11: |
RPM package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2.mira2.git.b0dd5db.4d22b5fChangeset:
project: openstack/neutron branch: openstack-ci/fuel-5.1.2/2014.1.1 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: patchset-createdFiles placed on repository:
openstack-neutron-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-bigswitch-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-brocade-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-cisco-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-hyperv-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-ibm-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-linuxbridge-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-mellanox-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-metaplugin-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-metering-agent-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-midonet-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-ml2-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-nec-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-nuage-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-ofagent-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-oneconvergence-nvsd-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-openvswitch-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-plumgrid-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-ryu-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-vmware-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm openstack-neutron-vpn-agent-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpm python-neutron-2014.1.3-fuel5.1.2.mira2.git.b0dd5db.4d22b5f.noarch.rpmNOTE: Changeset is not merged, created temporary package repository.
RPM repository URL:| wrote on 2014-12-11: |
(3.4 KiB)
DEB package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2~mira2+git.b0dd5db.4d22b5fChangeset:
project: openstack/neutron branch: openstack-ci/fuel-5.1.2/2014.1.1 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: patchset-createdFiles placed on repository:
neutron-common_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-dhcp-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-l3-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-lbaas-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-metadata-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-metering-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-bigswitch-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-bigswitch_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-brocade_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-cisco_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-hyperv_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-ibm-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-ibm_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-linuxbridge-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-linuxbridge_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-metaplugin_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-metering-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-midonet_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-ml2_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-mlnx-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-mlnx_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-nec-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-nec_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-nicira_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-oneconvergence-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-oneconvergence_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-openflow-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-openvswitch-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-openvswitch_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-plumgrid_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-ryu-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-ryu_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-vmware_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-plugin-vpn-agent_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-server_2014.1.3-fuel5.1.2~mira2+git.b0dd5db.4d22b5f_all.deb neutron-vpn-agent_2014.1.3-fuel5.1.2~mira2+git.b0...
| wrote on 2014-12-15: |
RPM package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2.mira2Changeset:
project: openstack/neutron branch: openstack-ci/fuel-5.1.2/2014.1.1 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: change-mergedFiles placed on repository:
openstack-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-bigswitch-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-brocade-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-cisco-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-hyperv-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-ibm-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-linuxbridge-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-mellanox-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-metaplugin-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-metering-agent-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-midonet-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-ml2-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-nec-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-nuage-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-ofagent-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-oneconvergence-nvsd-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-openvswitch-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-plumgrid-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-ryu-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-vmware-2014.1.3-fuel5.1.2.mira2.noarch.rpm openstack-neutron-vpn-agent-2014.1.3-fuel5.1.2.mira2.noarch.rpm python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpmChangeset merged. Package placed on primary repository
RPM repository URL:| wrote on 2014-12-15: |
DEB package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2~mira2Changeset:
project: openstack/neutron branch: openstack-ci/fuel-5.1.2/2014.1.1 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: change-mergedFiles placed on repository:
neutron-common_2014.1.3-fuel5.1.2~mira2_all.deb neutron-dhcp-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-l3-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-lbaas-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-metadata-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-metering-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-bigswitch-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-bigswitch_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-brocade_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-cisco_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-hyperv_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-ibm-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-ibm_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-linuxbridge-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-linuxbridge_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-metaplugin_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-metering-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-midonet_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-ml2_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-mlnx-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-mlnx_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-nec-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-nec_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-nicira_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-oneconvergence-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-oneconvergence_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-openflow-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-openvswitch-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-openvswitch_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-plumgrid_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-ryu-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-ryu_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-vmware_2014.1.3-fuel5.1.2~mira2_all.deb neutron-plugin-vpn-agent_2014.1.3-fuel5.1.2~mira2_all.deb neutron-server_2014.1.3-fuel5.1.2~mira2_all.deb neutron-vpn-agent_2014.1.3-fuel5.1.2~mira2_all.deb python-neutron_2014.1.3-fuel5.1.2~mira2_all.debChangeset merged. Package placed on primary repository
DEB repository URL:| wrote on 2014-12-22: |
Release note for 5.1.1:
ISSUE: Virtual Machines fail to start due to slow processing of IP tables rules
SUMMARY: In some larger scale environments, virtual machines fail to launch due to slow processing of IP tables rules. Environments with large numbers of compute nodes and VMs can lead to large IP tables in total if they apllied. Slow processing of large IP tables can lead to service timeouts that may prevent virtual machines from launching.
The problem is fixed in python-neutron package from MOS 5.1.2 (attached to the LP ticket).
Steps to apply patches on CentOS environments:
(should be done one by one on all compute nodes, then on all controllers, also one by one)Upload package python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm to current node:
Execute the following on the node: Run the following commands: [root@node-2 ~]# rpm -Uvh --nodeps python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm warning: python-neutron-2014.1.3-fuel5.1.2.mira2.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID bf906159: NOKEY Preparing... ########################################### [100%] 1:python-neutron ########################################### [100%] Make sure step 2 doesn’t produce any errors On computes nodes do: service neutron-openvswitch-agent restart On one of controller nodes: crm resource restart clone_p_neutron-openvswitch-agent| wrote on 2014-12-29: |
If it was a Neutron bug, how is it possible that this had to be fixed in 5.1.x but doesn't reproduce in 5.0.x?
| wrote on 2014-12-29: |
We also still need a backport of this for 4.1.x, status changed from Won't Fix to Confirmed.
| wrote on 2015-01-26: |
No updates since December, what's the status of the backports?
| wrote on 2015-01-29: |
For this patch which affects 4.x and 5.x versions Mirantis published official Tech Bulletin:
Consider to move this bug to Won't fix.
To post a comment you must .
This report contains Public information
Everyone can see this information.
You are
RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira13.git.f90d4c0.a299407Changeset:
project: openstack/neutron branch: openstack-ci/fuel-6.0/2014.2 author: Alexander Ignatov committer: Alexander Ignatov subject: Improve the performance of _modify_rules() in IptablesManager status: patchset-createdFiles placed on repository:
openstack-neutron-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-bigswitch-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-brocade-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-cisco-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-hyperv-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-ibm-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-linuxbridge-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-mellanox-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-metaplugin-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-metering-agent-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-midonet-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-ml2-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-nec-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-nuage-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-ofagent-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-oneconvergence-nvsd-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-opencontrail-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-openvswitch-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-plumgrid-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-ryu-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-vmware-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm openstack-neutron-vpn-agent-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpm python-neutron-2014.2-fuel6.0.mira13.git.f90d4c0.a299407.noarch.rpmNOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: