Skip to content

ipv6: set default_egress_policy for ingress rules - #6364

Merged
nvazquez merged 1 commit into
apache:mainfrom
weizhouapache:4.17-ipv6-fix-egress-rule-drop
May 7, 2022
Merged

nvazquez merged 1 commit into
apache:mainfrom
weizhouapache:4.17-ipv6-fix-egress-rule-drop

Conversation

@weizhouapache

@weizhouapache weizhouapache commented May 6, 2022

Copy link
Copy Markdown
Member

Description

This PR fixes an issue we found in the smoke test test/integration/smoke/test_network_ipv6.py.
sometimes the test failed with error below

FAIL: Test to verify IPv6 network
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/marvin/lib/decoratorGenerators.py", line 30, in test_wrapper
    return test(self, *args, **kwargs)
  File "/marvin/test_network_ipv6.py", line 1215, in test_01_verify_ipv6_network
    self.checkNetworkRouting()
  File "/marvin/test_network_ipv6.py", line 1060, in checkNetworkRouting
    "Ping from VM %s of network %s to VM %s of network %s is unsuccessful" % (self.routing_test_vm.id, self.routing_test_network.id, self.virtual_machine.id, self.network.id))
AssertionError: False is not true : Ping from VM 0aa36a76-09c6-476f-97c5-b9cea27a5b7c of network 27a2b244-e319-46c5-a779-d6ae73eb9ac2 to VM ae13ea17-1f35-4ca7-83c1-e13126f8df79 of network 1f38a686-69f3-41ed-a75e-cd3f822497d8 is unsuccessful

After investigation, we found the egress traffic is dropped by nft.

a correct nft chain looks like

root@r-282-VM:~# nft list chain ip6 ip6_firewall fw_chain_egress
table ip6 ip6_firewall {
	chain fw_chain_egress {
		counter packets 0 bytes 0 accept
	}
}

However, some VRs have the following nft chain

root@r-280-VM:~# nft list chain ip6 ip6_firewall fw_chain_egress
table ip6 ip6_firewall {
	chain fw_chain_egress {
		counter packets 0 bytes 0 drop
	}
}

It is because the ingress rule does not have correct default_egress_policy

root@r-280-VM:~# cat /etc/cloudstack/ipv6firewallrules.json
{
  "0": {
    "already_added": false,
    "default_egress_policy": true,
    "dest_cidr_list": [],
    "guest_ip6_cidr": "fd17:ac56:1234:1a96::/64",
    "id": 0,
    "protocol": "all",
    "purpose": "Ipv6Firewall",
    "revoked": false,
    "source_cidr_list": [],
    "src_ip": "",
    "traffic_type": "Egress"
  },
  "1263": {
    "already_added": false,
    "default_egress_policy": false,
    "dest_cidr_list": [
      "::/0"
    ],
    "guest_ip6_cidr": "fd17:ac56:1234:1a96::/64",
    "icmp_code": -1,
    "icmp_type": -1,
    "id": 1263,
    "protocol": "icmp",
    "purpose": "Ipv6Firewall",
    "revoked": false,
    "source_cidr_list": [
      "::/0"
    ],
    "traffic_type": "Ingress"
  },
  "id": "ipv6firewallrules"
}

in mose time, the Egress rule is processed before Ingress rule.
But when the Ingress rule is processed at first, the nft chain will be wrong.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

The issue is found in the smoke test `test/integration/smoke/test_network_ipv6.py`.
sometimes the test failed with error below
```
FAIL: Test to verify IPv6 network
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/marvin/lib/decoratorGenerators.py", line 30, in test_wrapper
    return test(self, *args, **kwargs)
  File "/marvin/test_network_ipv6.py", line 1215, in test_01_verify_ipv6_network
    self.checkNetworkRouting()
  File "/marvin/test_network_ipv6.py", line 1060, in checkNetworkRouting
    "Ping from VM %s of network %s to VM %s of network %s is unsuccessful" % (self.routing_test_vm.id, self.routing_test_network.id, self.virtual_machine.id, self.network.id))
AssertionError: False is not true : Ping from VM 0aa36a76-09c6-476f-97c5-b9cea27a5b7c of network 27a2b244-e319-46c5-a779-d6ae73eb9ac2 to VM ae13ea17-1f35-4ca7-83c1-e13126f8df79 of network 1f38a686-69f3-41ed-a75e-cd3f822497d8 is unsuccessful
```

After investigation, we found the egress traffic is dropped by `nft`.

a correct nft chain looks like
```
root@r-282-VM:~# nft list chain ip6 ip6_firewall fw_chain_egress
table ip6 ip6_firewall {
	chain fw_chain_egress {
		counter packets 0 bytes 0 accept
	}
}
```

However, some VRs has the following nft chain
```
root@r-280-VM:~# nft list chain ip6 ip6_firewall fw_chain_egress
table ip6 ip6_firewall {
	chain fw_chain_egress {
		counter packets 0 bytes 0 drop
	}
}
```

It is because the ingress rule does not have correct `default_egress_policy`
```
root@r-280-VM:~# cat /etc/cloudstack/ipv6firewallrules.json
{
  "0": {
    "already_added": false,
    "default_egress_policy": true,
    "dest_cidr_list": [],
    "guest_ip6_cidr": "fd17:ac56:1234:1a96::/64",
    "id": 0,
    "protocol": "all",
    "purpose": "Ipv6Firewall",
    "revoked": false,
    "source_cidr_list": [],
    "src_ip": "",
    "traffic_type": "Egress"
  },
  "1263": {
    "already_added": false,
    "default_egress_policy": false,
    "dest_cidr_list": [
      "::/0"
    ],
    "guest_ip6_cidr": "fd17:ac56:1234:1a96::/64",
    "icmp_code": -1,
    "icmp_type": -1,
    "id": 1263,
    "protocol": "icmp",
    "purpose": "Ipv6Firewall",
    "revoked": false,
    "source_cidr_list": [
      "::/0"
    ],
    "traffic_type": "Ingress"
  },
  "id": "ipv6firewallrules"
}
```

in mose time, the Egress rule is processed before Ingress rule.
But when the Ingress rule is processed at first, the nft chain will be wrong.
@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@weizhouapache weizhouapache added this to the 4.17.0.0 milestone May 6, 2022
@weizhouapache
weizhouapache requested review from nvazquez and shwstppr May 6, 2022 18:14
@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3362

@sonarqubecloud

sonarqubecloud Bot commented May 6, 2022

Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3363

@nvazquez

nvazquez commented May 6, 2022

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian Build Failed (tid-4130)

@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan test matrix

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan test centos7 vmware-70u3

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + vmware-70u3) has been kicked to run smoke tests

@apache apache deleted a comment from blueorangutan May 6, 2022
@apache apache deleted a comment from blueorangutan May 6, 2022
@apache apache deleted a comment from blueorangutan May 6, 2022
@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan test ubuntu20 kvm-ubuntu20

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

@apache apache deleted a comment from blueorangutan May 7, 2022
@apache apache deleted a comment from blueorangutan May 7, 2022
@apache apache deleted a comment from blueorangutan May 7, 2022
@apache apache deleted a comment from blueorangutan May 7, 2022
@apache apache deleted a comment from blueorangutan May 7, 2022
@blueorangutan

Copy link
Copy Markdown

Trillian Build Failed (tid-4141)

@blueorangutan

Copy link
Copy Markdown

Trillian Build Failed (tid-4143)

@nvazquez
nvazquez marked this pull request as ready for review May 7, 2022 03:23

@shwstppr shwstppr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Ran smoke test from #6338 to test different scenarios in addition to manually testing IPv6 Firewall rules

Test to verify IPv6 network ... === TestName: test_01_verify_ipv6_network | Status : SUCCESS ===
ok
Test to verify redundant IPv6 network ... === TestName: test_02_verify_ipv6_network_redundant | Status : SUCCESS ===
ok
Test to verify IPv4 network upgraded to IPv6 network ... === TestName: test_03_verify_upgraded_ipv6_network | Status : SUCCESS ===
ok
Test to verify redundant IPv4 network upgraded to redundant IPv6 network ... === TestName: test_04_verify_upgraded_ipv6_network_redundant | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 4 tests in 1694.090s

OK

@blueorangutan

Copy link
Copy Markdown

Trillian Build Failed (tid-4146)

@nvazquez
nvazquez merged commit c1691a9 into apache:main May 7, 2022
@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-4144)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39935 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6364-t4144-xenserver-71.zip
Smoke tests completed. 95 look OK, 1 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
test_04_extract_Iso Failure 128.31 test_iso.py

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-4145)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40636 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6364-t4145-kvm-centos7.zip
Smoke tests completed. 96 look OK, 0 have errors
Only failed tests results shown below:

Test Result Time (s) Test File

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-4142)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44625 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6364-t4142-vmware-65u2.zip
Smoke tests completed. 94 look OK, 2 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
test_02_verify_ipv6_network_redundant Failure 191.96 test_network_ipv6.py
test_01_redundant_vpc_site2site_vpn Failure 729.13 test_vpc_vpn.py

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-4148)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 37877 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6364-t4148-kvm-ubuntu20.zip
Smoke tests completed. 96 look OK, 0 have errors
Only failed tests results shown below:

Test Result Time (s) Test File

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-4147)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49743 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6364-t4147-vmware-70u3.zip
Smoke tests completed. 95 look OK, 1 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
test_02_verify_ipv6_network_redundant Failure 182.65 test_network_ipv6.py

@weizhouapache
weizhouapache deleted the 4.17-ipv6-fix-egress-rule-drop branch December 9, 2022 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants