SSVM: 'allow from' private IP in other SSVMs if the public IP is in allowed internal sites cidrs

[weizhouapache]

Description

This PR fixes #5678

Normally SSVMs in multiple zones communicate via the public IPs.
In some cases, the public IPs are in the internal allowed sites. There are some routes in SSVMs which causes the communication to use the private IPs. We need to add the private IP of other SSVM to /var/www/html/copy/.htaccess if the public IP is in secstorage.allowed.internal.sites

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

Without this PR, SSVM contains only public IP of another SSVM

root@s-8-VM:~# cat /var/www/html/copy/.htaccess
Options -Indexes
order deny,allow
deny from all
allow from 10.0.17.55

With this PR, it has both private and public IP of another SSVM

root@s-8-VM:~# cat /var/www/html/copy/.htaccess
Options -Indexes
order deny,allow
deny from all
allow from 10.0.17.55
allow from 10.0.40.129

The SSVM has route to all internal allowed sites via private nic, for example (secstorage.allowed.internal.sites=10.0.0.0/16)

root@s-8-VM:~# ip route
default via 10.0.96.1 dev eth2 
8.8.4.4 via 10.0.32.1 dev eth1 
10.0.0.0/16 via 10.0.32.1 dev eth1 
10.0.32.0/20 dev eth1 proto kernel scope link src 10.0.36.127 
10.0.96.0/20 dev eth2 proto kernel scope link src 10.0.99.22 
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.148.119 

How Has This Been Tested?

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[codecov]

Codecov Report

Merging #7288 (a0e2d00) into 4.18 (aa881f6) will increase coverage by 0.24%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               4.18    #7288      +/-   ##
============================================
+ Coverage     12.69%   12.93%   +0.24%     
- Complexity     8655     8941     +286     
============================================
  Files          2716     2715       -1     
  Lines        256117   256120       +3     
  Branches      39927    39940      +13     
============================================
+ Hits          32503    33128     +625     
+ Misses       219482   218833     -649     
- Partials       4132     4159      +27     

Impacted Files	Coverage Δ
.../secondarystorage/SecondaryStorageManagerImpl.java	4.65% <0.00%> (-0.09%)	⬇️

... and 128 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[harikrishna-patnala]

@blueorangutan package

[blueorangutan]

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with

SystemVM template(s). I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5658

[harikrishna-patnala]

@blueorangutan test centos7 kvm-centos7 qemuEv keepEnv

[blueorangutan]

@harikrishna-patnala a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian Build Failed (tid-6241)

[DaanHoogland]

clgtm

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[shwstppr]

Code LGTM

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6059

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-6555)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49594 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7288-t6555-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	Failure	499.76	test_kubernetes_clusters.py

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6494

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-7116)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45638 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7288-t7116-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	Error	77.61	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	Error	57.44	test_vm_life_cycle.py

[DaanHoogland]

verified in a 3-zone env, merging

[weizhouapache]

verified in a 3-zone env, merging

cool, thanks for testing @DaanHoogland

[daviftorres]

Dear @weizhouapache,

I couldn't find any information why this route is necessary: The SSVM has route to all internal allowed sites via private nic, for example (secstorage.allowed.internal.sites=10.0.0.0/16)

In my setup, both the management server cluster and all System VMs are behind a reverse proxy. I use the secstorage.allowed.internal.sites setting to whitelist these reverse proxies.

The reverse proxy serves multiple purposes: WAF, rate-limiting abusive users, DoS/DDoS protection, SSL/TLS termination, etc. It is deployed in the public network, directly connected to the System VMs.

However, when the SSVM creates a route via the private (management) network, this results in asymmetric routing, as requests come from the public network but replies are sent via the private NIC.

Could you clarify the rationale behind enforcing this route via the private NIC? Is it strictly necessary for the SSVM to behave this way?

I appreciate your time and insights.

Best regards,

[weizhouapache]

@daviftorres
it is a quite old PR, let me try to recall what the issue was. we had an issue in the past that template cannnot be copied from a zone to another zone. refer to #5678
this PR changes the file /var/www/html/copy/.htaccess as the SSVMs connect to each other via the private IPs. With this PR, both Public IP and private IP are allowed, according to .htaccess

Did you question the route or the configuration (/var/www/html/copy/.htaccess) ? This PR did not touch the routes inside SSVM.

The SSVM has route to all internal allowed sites via private nic, for example (secstorage.allowed.internal.sites=10.0.0.0/16)

this is by design I think. If it does not work in your environment, you can unset secstorage.allowed.internal.sites

[daviftorres]

My question was about the route. I can confirm when I set secstorage.allowed.internal.sites it creates a route next time the SSVM gets deployed. That is why I had to unset it. But at the same time, I cannot "allow" the reverse proxy IP.

[daviftorres]

@weizhouapache First time I set secstorage.allowed.internal.sites it fixed my problem of copying between zones. But minutes later the SSVM was replaced and the new route created asymmetric route.

[weizhouapache]

@weizhouapache First time I set secstorage.allowed.internal.sites it fixed my problem of copying between zones. But minutes later the SSVM was replaced and the new route created asymmetric route.

@daviftorres
Maybe you can test secstorage.allowed.internal.sites with other values
It can be set to multiple smaller cidrs separated by comma

[daviftorres]

@weizhouapache

Yes, I did. But my reverse proxies are on the public network, and since a route is being created that forces traffic through the management network, the issue happens. I’m still trying to understand why that route is added.

I know SysVMs aren’t officially meant to be behind a reverse proxy, but I’ve been doing it successfully for six months. Everything works fine except for this routing issue, which affects copying between zones.

If you want I can elaborate on it for better official support.

[weizhouapache]

Yes, I did. But my reverse proxies are on the public network, and since a route is being created that forces traffic through the management network, the issue happens. I’m still trying to understand why that route is added.

I think routes are configured based on that internal traffic goes through the private network, while public traffic goes through the public network. it makes sense for me.

I know SysVMs aren’t officially meant to be behind a reverse proxy, but I’ve been doing it successfully for six months. Everything works fine except for this routing issue, which affects copying between zones.

You can configure internal cidrs which does not contain the reverse proxy IP. I guess the route might not be an issue then.
can you give more details ?

[daviftorres]

@weizhouapache ,

I have this issue reported here: #5678

Since SSVM serves client's requests for downloading and uploading ISOs, Templates, etc on the Public network:

As you can see, the traffic that comes from the Public is responded via the Private (Management) network, breaking communication.

What am I miss understanding from the feature provided by the setting secstorage.allowed.internal.sites?

[weizhouapache]

@weizhouapache ,

I have this issue reported here: #5678

Since SSVM serves client's requests for downloading and uploading ISOs, Templates, etc on the Public network:

As you can see, the traffic that comes from the Public is responded via the Private (Management) network, breaking communication.

What am I miss understanding from the feature provided by the setting secstorage.allowed.internal.sites?

@daviftorres
I think you have answered the questions in #11020 , right ?

IMHO, there is no issue with routes, or .htaccess for setting secstorage.allowed.internal.sites in SSVM.
let's discuss in #11020

[daviftorres]

@weizhouapache ,

You are absolutely right. The two topics that I am bringing to discussion are directly related but not the same.

In #11020, I proposed to make the proxy.cidr, proxy.header.names, and proxy.header.verify to add entries in /var/www/html/copy/.htaccess.

In this conversations here, I am trying to understand why the route is created because I do not understand its purpose and it ends up breaking my setup (when using secstorage.allowed.internal.sites).

But they both compliment each other.