Skip to content

test(orchestrator): adopt per-test logger contract outside pxeboot - #5496

Open
balajikumaran-c-s wants to merge 2 commits into
dell:pub/rhel10.2from
balajikumaran-c-s:orch-test-split-2
Open

test(orchestrator): adopt per-test logger contract outside pxeboot#5496
balajikumaran-c-s wants to merge 2 commits into
dell:pub/rhel10.2from
balajikumaran-c-s:orch-test-split-2

Conversation

@balajikumaran-c-s

@balajikumaran-c-s balajikumaran-c-s commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

PR Title

test(orchestrator): adopt per-test logger contract outside pxeboot

Description of the Solution

This is part 2 of 3 of the Orchestrator test automation update. It moves the precheck, prepare, provision, cleanup and NFT tests to the per-test logger contract and their suite order blocks, and adds the OIM negative precheck cases. Each test resolves its registry entry and creates its own TestLogger; the result helpers only render the final record and enforce it.

Related Issue

Depends on part 1 (#5495) and contains its commit. Until part 1 merges, review only the second commit here (test(orchestrator): adopt per-test logger contract outside pxeboot). Part 3 (#5497) builds on this branch.

Changes

Result contract (test/orchestrator/fvt/result.py, test/orchestrator/nft/result.py)

  • verify_precheck, verify_cleanup and verify_nft take the test's TestLogger and registry case instead of a registry key.
  • New verify_precheck_rejection passes only when a precheck rejects an invalid input with an error.
  • Optional capabilities skip through one shared helper that records the skip reason.
  • verify_pxeboot keeps its (host, key, checker) form in this PR so the pxeboot suites are unchanged; part 3 moves it to the same contract.

Precheck (test/orchestrator/fvt/precheck)

  • OIM negative cases: CPU, memory and disk thresholds, a missing admin NIC, PXE/public NIC overlap, air-gapped internet, an unreachable SSH target, and an OS mismatch (ORCH_FVT_PRECHECK_V110-V117).
  • Storage negative cases carry function-level negative and mount_config markers instead of a module pytestmark that also marked them sanity.
  • Orders move into the precheck blocks (10000-10500).

Prepare, provision and cleanup (test/orchestrator/fvt/prepare, provision, cleanup)

  • Tests adopt the contract and order blocks (prepare 20000, provision 30000, cleanup 50000).
  • Provision boot image checks carry a boot_image marker instead of functional, so they run when no marker is selected.
  • The prepare OpenLDAP suite is unchanged here; part 3 moves its external LDAP cases to pxeboot.

NFT (test/orchestrator/nft)

  • NFT tests adopt the contract and the 60000 order block.

Documentation (test/orchestrator/fvt/README.md, test/orchestrator/nft/README.md)

  • The FVT README documents the order blocks, the renumbered IDs, and the full test inventory; the NFT README documents its order block.

Files Changed

File Change Type Description
test/orchestrator/fvt/README.md Modified Order blocks, renumbered IDs, test inventory
test/orchestrator/fvt/result.py Modified Per-test logger contract, rejection helper
test/orchestrator/fvt/cleanup/artifacts/test_artifacts.py Modified Contract and order block
test/orchestrator/fvt/cleanup/credentials/test_credentials.py Modified Contract and order block
test/orchestrator/fvt/cleanup/kubernetes/test_kubernetes.py Modified Contract and order block
test/orchestrator/fvt/cleanup/openchami/test_openchami.py Modified Contract and order block
test/orchestrator/fvt/cleanup/openldap/test_openldap.py Modified Contract and order block
test/orchestrator/fvt/cleanup/slurm/test_slurm.py Modified Contract and order block
test/orchestrator/fvt/cleanup/test_playbook.py Modified Root order
test/orchestrator/fvt/precheck/dependencies/test_dependencies.py Modified Contract and order block
test/orchestrator/fvt/precheck/environment/test_environment.py Modified Contract and order block
test/orchestrator/fvt/precheck/inputs/test_inputs.py Modified Contract and order block
test/orchestrator/fvt/precheck/oim_readiness/test_oim_hardware.py Modified Contract, CPU/memory/disk negative cases
test/orchestrator/fvt/precheck/oim_readiness/test_oim_network.py Modified Contract, missing NIC and NIC overlap negative cases
test/orchestrator/fvt/precheck/oim_readiness/test_oim_prerequisites.py Modified Contract, air-gapped internet, SSH and OS negative cases
test/orchestrator/fvt/precheck/storage/test_storage.py Modified Contract and order block
test/orchestrator/fvt/precheck/storage/test_storage_config_validation.py Modified Contract, function-level negative markers
test/orchestrator/fvt/precheck/test_playbook.py Modified Root order
test/orchestrator/fvt/prepare/network/test_network.py Modified Order block
test/orchestrator/fvt/prepare/openchami/test_openchami.py Modified Order block
test/orchestrator/fvt/prepare/test_playbook.py Modified Root order
test/orchestrator/fvt/provision/openchami/test_provision_state.py Modified Contract, order block, boot_image marker
test/orchestrator/fvt/provision/test_playbook.py Modified Root order
test/orchestrator/nft/README.md Modified NFT order block
test/orchestrator/nft/result.py Modified Per-test logger contract
test/orchestrator/nft/test_idempotency.py Modified Contract and order block
test/orchestrator/nft/test_lifecycle.py Modified Contract and order block
test/orchestrator/nft/test_performance.py Modified Contract and order block
test/orchestrator/nft/test_security.py Modified Contract and order block

Testing

  • python3 -m pytest --confcutdir=ut ut -q (from test/orchestrator): 143 passed.
  • python3 -m pytest fvt nft --collect-only -q: 311 tests collected, no collection errors. Every pxeboot caller (135) still uses the (host, key, checker) form.
  • The transitional verify_pxeboot was exercised with stub checkers: pass, skip (reason recorded) and failure (assertion raised).
  • Not run: FVT/NFT against a cluster for this branch.

Backward Compatibility

  • No breaking changes to playbooks, inputs or other domains.
  • Test case IDs, titles and markers follow the registry from part 1; the provision boot image checks now run without a marker.
  • External callers of verify_precheck, verify_cleanup or verify_nft must pass the TestLogger and case; all callers in the repository are updated.

Suggested Reviewers

Reviewer Review Focus
@sujit-jadhav Orchestrator precheck, prepare, provision and cleanup suites
@abhishek-sa1 Result contract and NFT

@balajikumaran-c-s
test(orchestrator): update verification library and unit tests
b8b6878 
Prepare the shared layer that the lifecycle suites build on.

- omnia_auto lists skip reasons under the summary table, drops the
  expected/unexpected/framework skip classification, and the wheel is
  rebuilt. add_session_result takes an optional reason, so other domains
  are unchanged.
- The test case registry gains the OIM negative cases, the local etcd
  media case, renumbered duplicate IDs, and per-suite order blocks.
- The verification library adds local etcd media, control-plane reboot
  recovery, OIM negative inputs, and LDAP identity gating. Node removal
  keeps the heredoc delimiter off the PXE mapping and needs at least two
  Slurm compute nodes. Recovery helpers check the reboot authorization.
  Host-key cleanup runs through run_on_host, and progress and cleanup
  messages use omnia_auto log. Unused imports are removed.
- Unit tests cover the etcd contracts, heredoc writes, the two-node
  removal minimum, host-key cleanup and progress logging.
- The test automation standard and test_config documentation describe
  the new rules and settings.

Refs dell#4849

Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
@balajikumaran-c-s
test(orchestrator): adopt per-test logger contract outside pxeboot
700c16b 
Each test now resolves its registry entry, creates its own TestLogger,
and passes both to verify_precheck, verify_precheck_rejection,
verify_cleanup or verify_nft. The result helpers only render the final
record. Tests move into their suite order blocks (precheck 10000,
prepare 20000, provision 30000, cleanup 50000, NFT 60000).

Precheck adds the OIM negative cases: CPU, memory and disk thresholds,
a missing admin NIC, PXE/public NIC overlap, air-gapped internet, an
unreachable SSH target and an OS mismatch. They use
verify_precheck_rejection, which passes only when the precheck rejects
the input with an error. The provision boot image checks carry a
boot_image marker so they run by default.

verify_pxeboot keeps its (host, key, checker) form for now so the pxeboot
suites are unchanged; the next change moves them to the same contract.
The FVT README lists the new order blocks and test inventory.

Refs dell#4849

Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com>
This was referenced Oct 9, 2026
Open
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abhishek-sa1 abhishek-sa1 abhishek-sa1 approved these changes
@VenkateswaraVatam VenkateswaraVatam Awaiting requested review from VenkateswaraVatam VenkateswaraVatam is a code owner
@RvishankarOMnia RvishankarOMnia Awaiting requested review from RvishankarOMnia RvishankarOMnia is a code owner
@snarthan snarthan Awaiting requested review from snarthan snarthan is a code owner
@sujit-jadhav sujit-jadhav Awaiting requested review from sujit-jadhav sujit-jadhav is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@balajikumaran-c-s @abhishek-sa1