Skip to content

[Test] Add to test RCA analysis: system messages and system journal. - #7594

Draft
gmarciani wants to merge 2 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3170/test-rca-0901-1
Draft

[Test] Add to test RCA analysis: system messages and system journal.#7594
gmarciani wants to merge 2 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3170/test-rca-0901-1

Conversation

@gmarciani

@gmarciani gmarciani commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Description of changes

Add to test RCA analysis: system messages and system journal.
Now when the cluster creation fails, the RCA utility returns the errors from both system messages and system journal.

This analysis would be helpful to troubleshoot cluster creation failures caused by transient system service failing to start. A recent observation was on NFS service failing to start without any clue about the root cause.

Tests

  1. Manual verification: injected a failure into the NFS start and verified that the journalctl command is able to identify the root cause.
[root@ip-27-6-45-152 ~]# sudo systemctl stop nfs-server
[root@ip-27-6-45-152 ~]# sudo mkdir -p /etc/systemd/system/nfs-mountd.service.d
printf '[Service]\nExecStartPre=/bin/false\n' | \
  sudo tee /etc/systemd/system/nfs-mountd.service.d/inject-fail.conf
sudo systemctl daemon-reload
[Service]
ExecStartPre=/bin/false
[root@ip-27-6-45-152 ~]# sudo systemctl start nfs-server
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.

[root@ip-27-6-45-152 ~]# sudo journalctl -b -p err --no-pager
...
Sep 01 19:22:26 ip-27-6-45-152 systemd[1]: Failed to start nfs-mountd.service - NFS Mount Daemon.
  1. SUCCESS integ test (happy path):
test-suites:
  basic:
    test_essential_features.py::test_essential_features:
      dimensions:
        - regions: [{{ US_EAST_1_INSTANCE_TYPE_0_xlarge_CAPACITY_RESERVATION_8_INSTANCES_2_HOURS_NOPG_DCV_OS_X86_1__g5g_2xlarge_CAPACITY_RESERVATION_1_INSTANCES_1_HOURS_NOPG_DCV_OS_X86_1 }}]
          instances: [{{ US_EAST_1_INSTANCE_TYPE_0 }}.xlarge]
          oss: [{{ DCV_OS_X86_1 }}]
          schedulers: ["slurm"]
  1. ONGOING Same integ test above but with injected failure to prove that the RCA automation is able to catch the error as expected

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@gmarciani gmarciani added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x Test labels Sep 1, 2026
Now when the cluster creation fails, the RCa utility returns the errors from both system messages and system journal.
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3170/test-rca-0901-1 branch from 0d58fbd to f05fd86 Compare September 1, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-changelog-update Disables the check that enforces changelog updates in PRs Test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant