cpvm: fix cpu usage for console vm when using vnc over websockets by alexandru-bagu · Pull Request #6970 · apache/cloudstack

alexandru-bagu · 2022-12-10T15:16:16Z

Description

On a VMware ESXi 7 environment the Console VM would always use 100% cpu after one connection and it would not stop until a VM restart (or well cloud service restart inside of the VM).

This PR fixes this issue by adding a Thread.sleep(1).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

sonarqubecloud · 2022-12-10T16:16:12Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
1 Security Hotspot
2 Code Smells

0.0% Coverage
0.0% Duplication

codecov · 2022-12-10T16:17:59Z

Codecov Report

Merging #6970 (0c85543) into 4.18 (66cbe0a) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               4.18    #6970   +/-   ##
=========================================
  Coverage     12.93%   12.93%           
  Complexity     8941     8941           
=========================================
  Files          2715     2715           
  Lines        256107   256107           
  Branches      39938    39938           
=========================================
  Hits          33128    33128           
  Misses       218820   218820           
  Partials       4159     4159

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

yadvr

LGTM, sleeping for 1ms; however I'm not sure if this could cause/introduce some kind of latency? cc @nvazquez

github-actions · 2023-01-27T13:08:09Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

sonarqubecloud · 2023-03-16T14:27:29Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

0.0% Coverage
0.0% Duplication

yadvr

Needs testing but otherwise LGTM

yadvr · 2023-04-19T07:41:39Z

cc @nvazquez
@blueorangutan package

blueorangutan · 2023-04-19T07:42:04Z

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-04-19T08:57:08Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5934

shwstppr · 2023-05-01T10:32:29Z

Tested this with a VMware 7 env and I'm not able to see any difference in the CPU usage.
This is what I did:

Setup a 4.18 env and started console on a couple of VMs. Soon CPU usage for CPVM was 100%
Upgraded env to this PR's packages and started console on a couple of VMs. CPU usage for CPVM again reached 100% within a minute or two.
After restarting cloud service on the CPVM CPU usage was <10%.

Not sure if we need to increase the sleep time or use a different approach. I'll try to test
@alexandru-bagu can you please have a look as well? Also, can you please rebase your change to 4.18 branch so that once we have a working fix it can go into 4.18.1

cc @rohityadavcloud @nvazquez

CPU usage from vCenter (Lows only when cloud service is restarted)

yadvr · 2023-05-23T05:29:34Z

cc @nvazquez to review
@blueorangutan package

blueorangutan · 2023-05-23T05:30:03Z

@rohityadavcloud a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-05-23T08:10:45Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6110

yadvr · 2023-08-08T18:32:36Z

@blueorangutan test

blueorangutan · 2023-08-08T18:34:06Z

@rohityadavcloud a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2023-08-08T18:36:22Z

@blueorangutan test alma8 vmware-7u3

blueorangutan · 2023-08-08T18:38:04Z

@rohityadavcloud [SF] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, alma9. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

yadvr · 2023-08-09T19:36:08Z

@blueorangutan test alma8 vmware-70u3

blueorangutan · 2023-08-09T19:38:03Z

@rohityadavcloud a [SF] Trillian-Jenkins test job (alma8 mgmt + vmware-70u3) has been kicked to run smoke tests

weizhouapache · 2023-08-11T08:12:51Z

@blueorangutan package

blueorangutan · 2023-08-11T08:14:03Z

@weizhouapache a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-11T09:09:12Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6749

yadvr · 2023-08-11T15:55:21Z

@blueorangutan test alma8 vmware-70u3

blueorangutan · 2023-08-11T15:56:03Z

@rohityadavcloud a [SF] Trillian-Jenkins test job (alma8 mgmt + vmware-70u3) has been kicked to run smoke tests

blueorangutan · 2023-08-12T04:33:09Z

[SF] Trillian test result (tid-7389)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server a8
Total time taken: 43568 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6970-t7389-vmware-70u3.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2023-08-14T06:49:37Z

                                updateFrontEndActivityTime();
                            }
                            connectionAlive = client.isVncOverWebSocketConnectionAlive();
+                            try {


a nested try should be in a separate method

weizhouapache · 2023-08-14T08:25:29Z

Tested this with a VMware 7 env and I'm not able to see any difference in the CPU usage. This is what I did:

Setup a 4.18 env and started console on a couple of VMs. Soon CPU usage for CPVM was 100%

Upgraded env to this PR's packages and started console on a couple of VMs. CPU usage for CPVM again reached 100% within a minute or two.

After restarting cloud service on the CPVM CPU usage was <10%.

Not sure if we need to increase the sleep time or use a different approach. I'll try to test @alexandru-bagu can you please have a look as well? Also, can you please rebase your change to 4.18 branch so that once we have a working fix it can go into 4.18.1

cc @rohityadavcloud @nvazquez

CPU usage from vCenter (Lows only when cloud service is restarted)

@shwstppr
thanks for the testing
do you think #7826 will fix the issue ?

weizhouapache · 2023-08-14T08:26:15Z

@alexandru-bagu
can you test #7826 ?

alexandru-bagu · 2023-08-14T08:35:48Z

I haven't had the CPU issue after I applied my own patch for it. I don't know how to trigger the cpu usage for either issues though.
Testing that commit would yield no new information for me.

weizhouapache · 2023-08-14T08:52:54Z

I haven't had the CPU issue after I applied my own patch for it. I don't know how to trigger the cpu usage for either issues though. Testing that commit would yield no new information for me.

ok , thanks @alexandru-bagu

@nvazquez @shwstppr @DaanHoogland @rohityadavcloud
do you think we should merge this #6970 or #7826 or both ? the vmware7 issue might be fixed by #7826 as well

DaanHoogland · 2023-08-14T09:43:58Z

@nvazquez @shwstppr @DaanHoogland @rohityadavcloud do you think we should merge this #6970 or #7826 or both ? the vmware7 issue might be fixed by #7826 as well

By the looks of the code both would help, but I do not know whether that would be overkill. Is it worth testing all four combinations of those two patches? cc @JoaoJandre @alexandru-bagu

yadvr · 2023-08-14T10:58:20Z

I've hit this issue on vmware7u3, now I no longer have the env; but this should be easy to test in our lab cc @weizhouapache @DaanHoogland

weizhouapache · 2023-08-14T13:43:59Z

verified ok

without this PR, the CPU usage increased to 100% even when only 1 vm console is open.
with this PR, the cpu usage is still low when 10 vm consoles are open

boring-cyborg Bot added the component:console-proxy label Dec 10, 2022

yadvr reviewed Dec 12, 2022

View reviewed changes

Comment thread services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxyNoVncClient.java Outdated

yadvr approved these changes Dec 12, 2022

View reviewed changes

weizhouapache requested a review from nvazquez December 12, 2022 18:20

stephankruggg approved these changes Dec 28, 2022

View reviewed changes

github-actions Bot added the status:has-conflicts label Jan 27, 2023

github-actions Bot removed the status:has-conflicts label Mar 16, 2023

alexandru-bagu force-pushed the console-vm-cpu-usage-fix branch from f208dc9 to bf8631a Compare March 16, 2023 13:24

yadvr added this to the 4.19.0.0 milestone Apr 19, 2023

yadvr approved these changes Apr 19, 2023

View reviewed changes

yadvr requested a review from shwstppr April 19, 2023 07:41

yadvr modified the milestones: 4.19.0.0, 4.18.1.0 Apr 20, 2023

yadvr assigned shwstppr and nvazquez Apr 20, 2023

yadvr changed the base branch from main to 4.18 April 28, 2023 10:51

boring-cyborg Bot added component:agent component:api component:build component:database component:orchestration component:storage labels Apr 28, 2023

yadvr mentioned this pull request Aug 8, 2023

cpvm: Fix CPVM not releasing CPU after closing console #7826

Merged

12 tasks

yadvr added the Severity:Critical Critical bug label Aug 8, 2023

DaanHoogland reviewed Aug 14, 2023

View reviewed changes

weizhouapache changed the title ~~fix cpu usage for console vm when using vnc over websockets~~ cpvm: fix cpu usage for console vm when using vnc over websockets Aug 14, 2023

weizhouapache approved these changes Aug 14, 2023

View reviewed changes

weizhouapache merged commit a47a4f4 into apache:4.18 Aug 14, 2023

JoaoJandre mentioned this pull request Aug 23, 2023

engine/schema: fix CPU cap limitation for upgraded environment #7808

Merged

12 tasks

weizhouapache mentioned this pull request Feb 21, 2024

Console Proxy VM: High CPU when using secure VNC / NIOConnection #8692

Closed

Conversation

alexandru-bagu commented Dec 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

sonarqubecloud Bot commented Dec 10, 2022

Uh oh!

codecov Bot commented Dec 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 27, 2023

Uh oh!

sonarqubecloud Bot commented Mar 16, 2023

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Apr 19, 2023

Uh oh!

blueorangutan commented Apr 19, 2023

Uh oh!

blueorangutan commented Apr 19, 2023

Uh oh!

shwstppr commented May 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented May 23, 2023

Uh oh!

blueorangutan commented May 23, 2023

Uh oh!

blueorangutan commented May 23, 2023

Uh oh!

yadvr commented Aug 8, 2023

Uh oh!

blueorangutan commented Aug 8, 2023

Uh oh!

yadvr commented Aug 8, 2023

Uh oh!

blueorangutan commented Aug 8, 2023

Uh oh!

yadvr commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

weizhouapache commented Aug 11, 2023

Uh oh!

blueorangutan commented Aug 11, 2023

Uh oh!

blueorangutan commented Aug 11, 2023

Uh oh!

yadvr commented Aug 11, 2023

Uh oh!

blueorangutan commented Aug 11, 2023

Uh oh!

blueorangutan commented Aug 12, 2023

Uh oh!

DaanHoogland Aug 14, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

weizhouapache commented Aug 14, 2023

Uh oh!

weizhouapache commented Aug 14, 2023

Uh oh!

alexandru-bagu commented Aug 14, 2023

Uh oh!

weizhouapache commented Aug 14, 2023

Uh oh!

DaanHoogland commented Aug 14, 2023

alexandru-bagu commented Dec 10, 2022 •

edited

Loading

codecov Bot commented Dec 10, 2022 •

edited

Loading

shwstppr commented May 1, 2023 •

edited

Loading