Conversation
* Store last successful MS host for agent reconnection fallback * Add newline at end of CapacityManagerImplTest.java
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.20 #14002 +/- ##
============================================
+ Coverage 16.32% 16.33% +0.01%
- Complexity 13556 13564 +8
============================================
Files 5669 5669
Lines 501399 501425 +26
Branches 60902 60904 +2
============================================
+ Hits 81847 81913 +66
+ Misses 410390 410341 -49
- Partials 9162 9171 +9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
it seems we have been working in the same area @Pearl1594 , #14000 is about agent rebalancing for local agents and #14001 about rebalancing remote agents. Can you have a look at those as well, please? |
|
It seems so - but they don't clash. I can have a look at them. Thanks @DaanHoogland |
|
|
@blueorangutan package |
|
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
DaanHoogland
moved this from Backlog
to conflict/waiting for author
in CloudStack Testing
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19030 |
There was a problem hiding this comment.
LGTM Tested manually with 2 ms
The parameter last.setup.completed.host is present in agent.properties
grep -n "^last.setup.completed.host" /etc/cloudstack/agent/agent.properties
Test A — the first-connect
1. systemctl stop cloudstack-agent
2. sed -i '/^last.setup.completed.host/d' /etc/cloudstack/agent/agent.properties
3.systemctl start cloudstack-agent
4. sleep 45
5. grep -c "^last.setup.completed.host" /etc/cloudstack/agent/agent.properties
Confirm the agent actually reached Ready before judging:
6. grep -n "Ready command is processed for agent" /var/log/cloudstack/agent/agent.log | tail -2
[root@ref-trl-12367-k-Mol8-kiran-chavala-kvm2 ~]# grep -n "Ready command is processed for agent" /var/log/cloudstack/agent/agent.log | tail -2
22778:2026-09-16 12:13:38,188 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:) Ready command is processed for agent [id: 2, uuid: 568a5362-6aa9-4b64-a9e7-786086ba2c93, name: ref-trl-12367-k-Mol8-kiran-chavala-kvm2]
22804:2026-09-16 12:13:39,613 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Ready command is processed for agent [id: 2, uuid: 568a5362-6aa9-4b64-a9e7-786086ba2c93, name: ref-trl-12367-k-Mol8-kiran-chavala-kvm2]
Test B — Break it centrally:
-
cmk list configurations name=host filter=name,value
-
cmk update configuration name=host value=10.99.99.99
-
Confirm propagation reached the agent:
MS side
[root@ref-trl-12367-k-Mol8-kiran-chavala-mgmt1 ~]# grep -n "Propagating management server list update to agents" /var/log/cloudstack/management/management-server.log
27017:2026-09-16 08:29:16,649 DEBUG [o.a.c.a.l.IndirectAgentLBServiceImpl] (qtp253011924-18:[ctx-72433d19, ctx-79e03618, ctx-882c796e]) (logid:86a3a74f) Propagating management server list update to agents
agent side
[root@ref-trl-12367-k-Mol8-kiran-chavala-kvm2 ~]#grep -n "Processed new management server list" /var/log/cloudstack/agent/agent.log
27000:2026-09-16 12:48:16,739 INFO [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:) Processed new management server list: 10.99.99.99@static
[root@ref-trl-12367-k-Mol8-kiran-chavala-kvm1 ~]# grep "^host=\|^last.setup.completed.host" /etc/cloudstack/agent/agent.properties
last.setup.completed.host=10.0.35.19
host=10.99.99.99@static
systemctl restart cloudstack-agent
tail -f /var/log/cloudstack/agent/agent.log
Expected sequence with the patch:
Connecting to host: 10.99.99.99 (Agent.java:332)
IOException while connecting to 10.99.99.99:8250 (NioClient.java:92)
Connecting to host: ← the appended fallback
Connected to the host: (Agent.java:662)
Ready command is processed for agent ...
The host returns to Up with no SSH intervention — that's the pass condition. Witn 10.99.99.99 indefinitely.
2026-09-16 12:51:11,843 DEBUG [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Multipath Cleanup Job elapsed time (ms): 30; result: 0
2026-09-16 12:51:23,321 ERROR [utils.nio.NioClient] (main:[]) (logid:) IOException while connecting to 10.99.99.99:8250 java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:591)
at java.base/sun.nio.ch.Net.connect(Net.java:598)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853)
at com.cloud.utils.nio.NioClient.init(NioClient.java:65)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112)
at com.cloud.agent.Agent.start(Agent.java:322)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
at com.cloud.agent.AgentShell.start(AgentShell.java:559)
at com.cloud.agent.AgentShell.main(AgentShell.java:589)
2026-09-16 12:51:23,323 WARN [utils.nio.NioClient] (main:[]) (logid:) Unable to connect to remote: is there a server running on port 8250? java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.Net.connect0(Native Method)
at java.base/sun.nio.ch.Net.connect(Net.java:591)
at java.base/sun.nio.ch.Net.connect(Net.java:598)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853)
at com.cloud.utils.nio.NioClient.init(NioClient.java:65)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112)
at com.cloud.agent.Agent.start(Agent.java:322)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
at com.cloud.agent.AgentShell.start(AgentShell.java:559)
at com.cloud.agent.AgentShell.main(AgentShell.java:589)
2026-09-16 12:51:23,324 WARN [cloud.agent.Agent] (main:[]) (logid:) Attempt to connect to server generated NIO Connection Exception Connection timed out, trying again
2026-09-16 12:51:28,326 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [ssl.handshake.timeout] has empty or null value. Using default value [null].
2026-09-16 12:51:28,326 INFO [cloud.agent.Agent] (main:[]) (logid:) Connecting to host: 10.0.35.171
2026-09-16 12:51:28,326 INFO [utils.nio.NioClient] (main:[]) (logid:) Connecting to 10.0.35.171:8250
2026-09-16 12:51:28,327 INFO [utils.nio.NioClient] (main:[]) (logid:) Connected to 10.0.35.171:8250
2026-09-16 12:51:28,330 INFO [utils.nio.Link] (main:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2026-09-16 12:51:28,872 INFO [utils.nio.NioClient] (main:[]) (logid:) SSL: Handshake done
|
@blueorangutan package |
|
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19274 |
|
@blueorangutan test |
|
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian Build Failed (tid-17020) |
This branch has not been deployed
6 participants
Description
The issue is that once a bad hosts configuration update made, there's no way to get the Agents back in Up state without SSH-ing on each of them and manually updating the agent.properties file (because they're not listening to any new update from a control plane, since they're not connected anymore).
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?