Sync pub/issue 4849 telemetry to staging by Kratika-P · Pull Request #5415 · dell/omnia

Kratika-P · 2026-09-30T12:02:48Z

PR Title

Description of the Solution

Related Issue

Fixes #

Changes

Component (`path/to/files`)

Change description

Files Changed

File	Change Type	Description
`path/to/file`	Added / Modified / Deleted	Brief description

Testing

Test or verification result

Backward Compatibility

Compatibility notes

Suggested Reviewers

Reviewer	Review Focus
`@reviewer-or-team`	Component, domain, security, test, or compatibility focus

… volumes (#5322) * updating precheck should be called everytime telemetry playbook is getting called Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * kafka cleanup for cluster id and secret cleanp only with delete_sinks_volume=true Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Revert "kafka cleanup for cluster id and secret cleanp only with delete_sinks_volume=true" This reverts commit 473c54f. Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(telemetry): preserve Kafka identity during cleanup Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * docs(repo_manager): align test READMEs with actual FVT, NFT, and UT test cases (#5283) The repo_manager test documentation was out of date with the actual test implementations. This commit: - Creates fvt/README.md as the authoritative FVT test-case registry (117 tests), following the image_build_manager FVT README format with per-phase tables, actual TC IDs (RM_FVT_*), markers, and pass criteria - Creates nft/README.md documenting all 5 non-functional tests (idempotency, performance, security) with RM_NFT_* IDs - Rewrites ut/README.md with the complete 141-test registry matching ut_test_case_vars.py (RM_UT_001 through RM_UT_143) - Updates docs/TEST_CASES.md to use actual source TC IDs instead of the former TC_RM_* convention, and adds NFT and catalog sections - Updates the top-level README.md with correct test counts (263 total), links to all sub-READMEs, NFT directory in the tree, and NFT markers Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix pipeline report attachment and repo_manager project_name configuration (#5281) - Remove test stage success check in GitLab CI pipeline to attach reports from failed test stages (repo_manager, orchestrator, telemetry) - Make project_name optional in repo_manager test_config.yml - Read project_name from OMNIA_PROJECT_NAME environment variable with fallback to "project_default" - Update validation to support optional project_name from env var This allows email notifications to include reports from failed test stages and enables flexible project_name configuration via environment variables. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * feat(test/discovery): add ut/ and nft/ folders aligned with image_build_manager (#5280) Align the discovery test folder structure with image_build_manager by adding the missing ut/ (unit test) and nft/ (non-functional test) directories and their test suites. Changes: - Add ut/ with 57 unit tests across three test files: - test_input_validation_schema.py: JSON schema and credential rules - test_discovery_config_validator.py: L2 semantic validator (OME IP) - test_standalone_independence.py: repo structure and independence - Add nft/ with performance and idempotency tests: - test_performance.py: precheck/execute/cleanup threshold tests - test_idempotency.py: precheck and cleanup idempotency tests - Add library/vars/ut_test_case_vars.py with stable DISC_UT_* IDs - Add NFT test cases (DISC_NFT_*) to test_case_vars.py - Update library/vars/__init__.py to export new variables - Update test_run_config.yml with nft_discovery and ut_discovery sections - Update conftest.py to register the nft marker - Update setup_env.sh: fix execute permission, update tab-completion for fvt_discovery/nft_discovery/ut_discovery categories - Fix run_validation.sh execute permission (100644 -> 100755) - Update README.md to document all three test categories Resolves #4849 Signed-off-by: saifudheene <saifudheen.e@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Add orchestrator FVT documentation and __init__.py Adds comprehensive FVT documentation matching the image_build_manager convention, including test-case registry, ID standard, directory layout, execution order, and command examples. Also adds __init__.py with scenario descriptions. Removes the redundant TEST_CASES.md redirect stub. This aligns orchestrator FVT structure with other domains and provides a single authoritative source for test-case information. Author: Sujit Jadhav <sujit.jadhav@dell.com> Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * BuildStream: support custom paths and sync all sample catalogs (#5279) * fix(build_stream): sync all sample catalogs to GitLab Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(test): honor custom Omnia paths in BuildStream Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(test): retain default BuildStream report path Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * test(orchestrator): align validation framework, runtime paths, and LDAP testing (#5285) * cleanup idempotent issue Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> * test(orchestrator): align runtime paths and LDAP validation Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> * test(orchestrator): satisfy NFT contract lint Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> * openldap user creation Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> * revert wheel file Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> --------- Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * feat: modernize Omnia CLI, catalog selection, and repository workflows (#5290) * feat(main): enhance CLI output and catalog management * ci: refine PR hygiene and ownership checks * feat(main): describe and select bundled catalogs * fix(orchestrator): standardize inventory filename * chore(telemetry): remove sample inventory header Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(catalog): resolve orphan image groups across RHEL 10.0/10.2 sample catalogs (#5291) Move image packages (ghcr_io/openchami/image_thrillhouse, docker_io/dellhpcomniaaisolution/image_build_el10, docker_io/dellhpcomniaaisolution/image_build_aarch64) into baseos_group and remove the separate image_thrillhouse_group, image_builder_x86_64_group, and image_builder_aarch64_group definitions and their functional layer references across all RHEL 10.0 and 10.2 sample catalogs. Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(orchestrator): harden resolver, mounts, and cleanup (#5276) * refactor(orchestrator): align credential artifact names Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Update BUILD_STREAM_DESIGN.md Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com> * fix(orchestrator): honor disabled VAST storage Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): retain legacy VAST input contract Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * chore(orchestrator): fix storage config formatting Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): strengthen input contract validation Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): preserve validation compatibility Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): harden validation and provisioning flows Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(telemetry): align OME FVT with Kafka Bridge TLS Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(discovery): standardize precheck and runtime paths Add a real Discovery precheck and shared OME endpoint validation. Honor DISCOVERY_DATA_PATH consistently across setup, validation, initialization, cleanup, outputs, and source documentation. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(discovery): require OME configuration directly Remove the legacy enable_bmc_discovery switch from the OME-only Discovery flow. Require and validate ome_ip consistently at L1 and L2, always collect OME credentials for selected credential flows, and always run the OME reachability precheck. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(discovery): standardize tagged workflows Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Fix SMD lifecycle and OIM mount handling Reconcile managed SMD identities, dependencies, group memberships, and metadata hostnames before registration and validate the final state. Refresh OpenCHAMI tokens for long-running provisioning, scope workload OIM mounts, and record validated mount ownership. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Separate SMD POC from OIM and resolver fixes Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): align persistent-state cleanup Remove the unused OIM mount ledger and delete OpenCHAMI persistent data volumes during cleanup. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): make cleanup redeploy-safe Remove installed OpenCHAMI packages during full cleanup, restore and verify the PostgreSQL initialization script for partial cleanup recovery, and honor cleanup_credentials=false when selecting cleanup components. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): prevent OpenCHAMI cleanup race Cancel pending OpenCHAMI target starts, remove containers retaining managed data volumes before volume deletion, and render the cleanup summary as structured output. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> --------- Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Remove OIM idrac IP variable from telemetry config and fix test case logic for victoria logs and metrics (#5287) * Remove OIM idrac IP variable from telemetry config Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry test case logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry test case logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry_func.py Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Create source_sink_mapping for telemetry_func.py Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry test case logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> --------- Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * feat(orchestrator): add etcd local disk test cases and remove hardcoded paths (#5294) Add comprehensive etcd local disk automation test cases Remove all hardcoded paths from orchestrator test automation to support custom installation paths via environment variables. ## Etcd Local Disk Test Cases (TC_K8_057-067) - TC_K8_057: Verify etcd_on_local_disk enabled in config - TC_K8_058: Verify BOSS card detection (skips if not detected) - TC_K8_059: Verify disk partitioning (GPT, not on root disk) - TC_K8_060: Verify filesystem creation (ext4) - TC_K8_061: Verify fstab and mount (UUID-based entries) - TC_K8_062: Verify local disk config (not NFS) - TC_K8_063: Verify fallback disk detection - TC_K8_064: Verify first boot setup (etcd-disk-setup.sh) - TC_K8_065: Verify SSD disk support (per-node detection) - TC_K8_066: Verify HDD disk support (per-node detection) - TC_K8_067: Verify NVMe disk support (per-node detection) ## Path Configuration Improvements - Add ENV_ORCHESTRATOR_DATA_PATH environment variable support - Add ENV_CATALOG_FILE_PATH environment variable support - Make SHARED_PATH dynamic with fallback to /opt/omnia - Update path templates to use {shared_path} placeholder - Add 3-tier priority for catalog path resolution: 1. CATALOG_FILE_PATH env var 2. orchestrator_config.yml catalog_file_path 3. Default template with OMNIA_DATA_PATH ## Configuration Options - CATALOG_FILE_PATH: Override catalog file path - OMNIA_DATA_PATH: Override Omnia base path - ORCHESTRATOR_DATA_PATH: Override orchestrator path Closes #4849 Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix BuildStream validate API runtime path handling (#4849) (#5292) * fix(test): honor custom Omnia paths in BuildStream Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(test): retain default BuildStream report path Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): align validate API runtime paths Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * test(orchestrator): expand BuildStream validation coverage Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): resolve validation report from test config Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix report naming and Run ID behavior for CLI and CI test runs (#5293) * Fix pipeline report attachment and repo_manager project_name configuration - Remove test stage success check in GitLab CI pipeline to attach reports from failed test stages (repo_manager, orchestrator, telemetry) - Make project_name optional in repo_manager test_config.yml - Read project_name from OMNIA_PROJECT_NAME environment variable with fallback to "project_default" - Update validation to support optional project_name from env var This allows email notifications to include reports from failed test stages and enables flexible project_name configuration via environment variables. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Update report naming to use test_config.yml report_name and remove timestamps - Modify build_report_name to extract domain_name from report_name in test_config.yml - Remove domain_name parameter from build_report_name function - Remove timestamp format from CLI report names - Use pipeline_id prefix for CI runs: <pipeline_id>_<report_name> - CLI runs now use report_name from config directly without timestamp - Update all conftest.py files to remove domain_name parameter - Update documentation to reflect new naming logic - Rebuild wheel with updated code This simplifies report naming and makes it more consistent with test configuration. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix report naming to use random IDs for CLI runs and remove timestamps - Remove timestamp-based report naming completely - Generate random 8-character UUID for CLI runs to avoid appending to same report - CI runs: <pipeline_id>_<base_name> format - CLI runs: <random_id>_<base_name> format (e.g., a1b2c3d4_omnia_main_test_report) - Remove unused _timestamp() function from validation_runner.py - Update build_report_name to use random IDs for CLI runs - Update documentation to reflect new naming logic - Rebuild wheel with updated code This ensures each CLI run creates a new report file instead of appending to existing ones. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix NameError: report_id not defined in validation_runner.py - Add report_id variable assignment in _run_fvt method after setting REPORT_ID env var - Add report_id variable assignment in _cmd_config method after setting REPORT_ID env var - Ensures report_id is available for logging and banner printing This fixes the NameError that occurred when running fvt_main verify command. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Use random Run IDs across all test domains and remove prefixes for CLI report filenames - build_report_name: CLI runs now return plain base_name from test_config.yml (no prefix) - build_report_name: CI runs keep <pipeline_id>_<base_name> format - Every domain conftest.py now generates a random 8-char UUID for report_id instead of reading REPORT_ID from environment (which was pipeline ID) - validation_runner.py: always generates a fresh random REPORT_ID for each run - Removed export REPORT_ID=$CI_PIPELINE_ID from .gitlab-ci-cluster.yml - Removed REPORT_ID seeding logic from build_stream/_run.py - Updated documentation in 06_report.md This ensures: 1. CLI report filenames are exactly as configured in test_config.yml 2. Re-running the same tests creates new Run IDs in the same report file 3. Runs no longer append/merge under pipeline ID Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix trailing spaces in .gitlab-ci-cluster.yml Removed trailing spaces introduced when removing REPORT_ID exports from the SSH command blocks. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add omnia repo commit ID to pipeline email notification - Added logic to get current commit ID of the omnia repository - Priority: CI_COMMIT_SHA (from GitLab CI) > git rev-parse HEAD (local) > unknown - Display commit ID in email body along with pipeline URL and trigger time - This helps track which omnia repository commit was used for the pipeline execution Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Rename report_id to run_id and fix timestamp format - Changed report_id to run_id in report_func.py, report_html.py, and all domain conftest.py files - Changed run_id timestamp format from YYYYMMDDHHmmSSfff to YYYYMMDD_HHMMSS - Fixed HTML report to display full run_id instead of truncated last 8 characters - Updated pipeline report collection to read report_name from test_config.yml and search for exact filenames - Updated all test_config.yml files to use run_id instead of report_id - Changed environment variable from REPORT_ID to RUN_ID - Rebuilt omnia_auto wheel with updated code Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix line length violation by shortening JSON filenames - Renamed domain_report_paths.json to report_paths.json - Renamed domain_report_names.json to report_names.json - This reduces line length from 166 to 159 characters Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Update remaining report_id references to run_id - Changed report_id to run_id in validation_runner.py - Changed REPORT_ID environment variable to RUN_ID - Updated VAST telemetry functions to use RUN_ID instead of REPORT_ID - Updated VAST trigger state file format to use run_id placeholder Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> --------- Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Refactor report_id to run_id and improve pipeline configuration (#5298) This commit consolidates multiple improvements to the test framework and pipeline: 1. Rename report_id to run_id throughout the codebase - Changed report_id to run_id in report_func.py, report_html.py, and all domain conftest.py files - Changed run_id timestamp format from YYYYMMDDHHmmSSfff to YYYYMMDD_HHMMSS - Fixed HTML report to display full run_id instead of truncated last 8 characters - Updated all test_config.yml files to use run_id instead of report_id - Changed environment variable from REPORT_ID to RUN_ID - Updated VAST telemetry functions to use RUN_ID instead of REPORT_ID - Made repo_manager follow standard report naming convention (removed category-based naming) - Rebuilt omnia_auto wheel with updated code 2. Improve pipeline report collection - Updated pipeline report collection to read report_name from test_config.yml - Fixed JSON filenames from domain_report_names.json to report_names.json (line length fix) - Updated cleanup stage order: telemetry -> orchestrator -> image_build_manager -> repo_manager - Updated dependencies to match new cleanup order - Added delete_sinks_volume=true to telemetry cleanup 3. Update email notification script - Updated stage order to match new cleanup order - Improved domain detection for test reports Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix default inventory, bmc group and pulp path resolution logic in telemetry (#5295) * Remove OIM idrac IP variable from telemetry config Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry test case logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry test case logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry_func.py Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Create source_sink_mapping for telemetry_func.py Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry test case logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Add default env path resolution in telemetry Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Update telemetry_validation.py Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix bmc_group_data.csv path resolution issue Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix for bmc_group_data_path resolution in telemetry Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix inventory and data path resolution default logic in telemetry test Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix telemetry_func.py logic Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Fix andible lint issues Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> * Path vulnerability fixes and parameterization of repo_url in telemetry Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> --------- Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(orchestrator): fix Kubernetes test cases skip logic and standardize test IDs (#5299) TC_K8_024 Fix: - Changed from /hsm/v2/groups (requires auth) to /hsm/v2/State/Components (public) - Test now checks for K8s nodes registered in SMD instead of functional groups - Eliminates authentication requirement and test skip condition - Shows node count and IDs in test output (e.g., 8 nodes: x1000c0s0b0n0, ...) - Updated test title and messages to reflect node-based verification PowerScale CSI Detection Fix (TC_K8_045, TC_K8_046, TC_K8_048): - Changed from catalog package check to enable_powerscale_csi attribute check - Now reads enable_powerscale_csi from omnia_config.yml - Tests skip when enable_powerscale_csi=false (configuration-based) - Previous catalog check was detecting PowerScale packages even when not configured - Test status: FAILED → SKIPPED (correct behavior) Etcd Skip Logic Fix (TC_K8_057, TC_K8_058, TC_K8_065, TC_K8_066, TC_K8_067): - Changed from tl.passed() to tl.skipped() when feature/hardware not available - TC_K8_057: Shows SKIP when etcd_on_local_disk disabled (was PASS) - TC_K8_058: Shows SKIP when BOSS card not detected (was PASS) - TC_K8_065: Shows SKIP when SSD not detected (was PASS) - TC_K8_066: Shows SKIP when HDD not detected (was PASS) - TC_K8_067: Shows SKIP when NVMe not detected (was PASS) - All tests now correctly show SKIP instead of PASS when feature not available Test ID Standardization: - Renumbered test IDs to remove gaps (TC_K8_050, TC_K8_055-056) - Sequential numbering from TC_K8_000 to TC_K8_064 (65 test cases total) - Updated test case IDs in k8s_vars.py, test files, and docstrings - No hardcoded paths, IPs, hostnames, or node names in any changes Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * (orchestrator): correct custom Slurm configuration guidance (#5301) * refactor(orchestrator): align credential artifact names Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Update BUILD_STREAM_DESIGN.md Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com> * fix(orchestrator): honor disabled VAST storage Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): retain legacy VAST input contract Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * chore(orchestrator): fix storage config formatting Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): strengthen input contract validation Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): preserve validation compatibility Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): harden validation and provisioning flows Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(telemetry): align OME FVT with Kafka Bridge TLS Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(discovery): standardize precheck and runtime paths Add a real Discovery precheck and shared OME endpoint validation. Honor DISCOVERY_DATA_PATH consistently across setup, validation, initialization, cleanup, outputs, and source documentation. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(discovery): require OME configuration directly Remove the legacy enable_bmc_discovery switch from the OME-only Discovery flow. Require and validate ome_ip consistently at L1 and L2, always collect OME credentials for selected credential flows, and always run the OME reachability precheck. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(discovery): standardize tagged workflows Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Fix SMD lifecycle and OIM mount handling Reconcile managed SMD identities, dependencies, group memberships, and metadata hostnames before registration and validate the final state. Refresh OpenCHAMI tokens for long-running provisioning, scope workload OIM mounts, and record validated mount ownership. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Separate SMD POC from OIM and resolver fixes Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): align persistent-state cleanup Remove the unused OIM mount ledger and delete OpenCHAMI persistent data volumes during cleanup. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): make cleanup redeploy-safe Remove installed OpenCHAMI packages during full cleanup, restore and verify the PostgreSQL initialization script for partial cleanup recovery, and honor cleanup_credentials=false when selecting cleanup components. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator): prevent OpenCHAMI cleanup race Cancel pending OpenCHAMI target starts, remove containers retaining managed data volumes before volume deletion, and render the cleanup summary as structured output. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * docs(orchestrator): clarify custom Slurm input nesting Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> --------- Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Update telemetry_func.py (#5300) Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix utils cleanup to remove status files and prevent recreation (#5302) * Fix utils domain cleanup and test automation output paths, add dataset templates This commit fixes several issues in the utils domain: 1. Fixed domain-init.sh cleanup to include output directory - Added output directory to cleanup paths - Resolves issue where status files were blocking full cleanup 2. Fixed test automation output path resolution - Created dedicated get_collect_output_path() function for collect scenario - Updated get_utils_output_path() to return base directory for status files - Updated collect tests to use new dedicated function - install_os, setup, and cleanup tests correctly use base directory 3. Added dataset generator templates for missing scenarios - Added backup_oim_logs_config.yml.j2 template - Added slurm_config_util_config.yml.j2 template - Updated defaults.yml with variables for both templates - Now supports dataset generation for all 4 utils scenarios These changes ensure: - Full cleanup works correctly via omnia.sh --cleanup --all - Test automation correctly locates output files for all scenarios - Dataset generation supports all utils domain input files Signed-off-by: SujalS27 <Sujal.Sooryavamshi@dell.com> * Update utils status writer version to 2.3.0 Fixed version mismatch in utils domain status file. - Changed version from 2.2.0 to 2.3.0 to match Utils Galaxy collection version - This is an informational change only, no functional impact Signed-off-by: SujalS27 <Sujal.Sooryavamshi@dell.com> * Fix utils cleanup to remove status files and prevent recreation The cleanup playbooks were recreating utils_status.yml after cleanup, causing --cleanup --all to fail because the output directory was not empty. Changes: - Modified utils.yml to skip status writer when cleanup tag is used - Added status file removal to all cleanup playbooks: - cleanup_logs.yml: removes utils_status.yml - cleanup_install_os.yml: removes utils_status.yml and install_os_status.yml - cleanup_backup_oim_logs.yml: removes utils_status.yml - cleanup_slurm_config_backups.yml: removes utils_status.yml - Removed status writer calls from cleanup playbook post_tasks This ensures cleanup leaves the output directory clean, allowing --cleanup --all to succeed. Signed-off-by: SujalS27 <Sujal.Sooryavamshi@dell.com> * Add test automation to verify status file removal during cleanup Added verification tests to ensure cleanup playbooks properly remove status files created by their corresponding main playbooks. Changes: - Added check_utils_status_file_removed() function - Added check_install_os_status_file_removed() function - Added test case cleanup_status_file_removed (UTILS_FVT_CLEANUP_V003) - Added test case cleanup_install_os_status_file_removed (UTILS_FVT_CLEANUP_INSTALL_OS_V004) - Added status file removal tests to all cleanup test files: - test_cleanup.py (combined cleanup) - test_cleanup_logs.py - test_cleanup_install_os.py - test_cleanup_backup_oim_logs.py - test_cleanup_slurm_config_backups.py All 20 cleanup tests pass successfully. Signed-off-by: SujalS27 <Sujal.Sooryavamshi@dell.com> --------- Signed-off-by: SujalS27 <Sujal.Sooryavamshi@dell.com> Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(telemetry): validate BMC data path for iDRAC Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * updating omnia.env with Omnia version as 2.3.0.0 Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(telemetry): harden deployment and cleanup flows Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> --------- Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Signed-off-by: saifudheene <saifudheen.e@dell.com> Signed-off-by: Sujit Jadhav <sujit.jadhav@dell.com> Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com> Signed-off-by: mithileshreddy04 <mithilesh.reddy@dell.com> Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> Signed-off-by: SujalS27 <Sujal.Sooryavamshi@dell.com> Co-authored-by: Katakam Rakesh Naga Sai <125246792+Katakam-Rakesh@users.noreply.github.com> Co-authored-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: saifudheene <Saifudheen.E@dell.com> Co-authored-by: Sujit Jadhav <sujit.jadhav@dell.com> Co-authored-by: SOWJANYAJAGADISH123 <Sowjanya.Jagadish@dell.com> Co-authored-by: Nagachandan P <Nagachandan.p@dell.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> Co-authored-by: balajikumaran.cs <balajikumaran.cs@dell.com> Co-authored-by: Mithilesh Reddy <mithilesh.reddy@dell.com> Co-authored-by: Sushmitha U <Sushmitha.U@dell.com> Co-authored-by: SujalS27 <Sujal.Sooryavamshi@dell.com>

…n flags and consolidate NFT test execution (#5338) * Add cleanup_credentials and cleanup_logs preservation flags to telemetry cleanup flow - Add cleanup_credentials and cleanup_logs variables to preserve credentials and logs during cleanup without volume deletion - Add 30-second warning when Delete_sinks_volume=true is combined with preservation flags, as credentials/logs are always deleted in volume mode - Support cleanup_credentials=false for both full cleanup (--tags cleanup) and granular cleanup (--tags cleanup_idrac, etc.) - Modify TEL_NFT_013 test_full_lifecycle to use cleanup_credentials=false for credential preservation during cleanup-then-redeploy cycle - Update cleanup resource summary to display credential and log preservation status - Update README.md files with new flag documentation and usage examples Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Modify Readme for telemetry nft - Remove test_nft_final_cluster_state_warning (TEL_NFT_019) from test_idempotency.py as it is not required as part of NFT suite - Removed corresponding test case entry from conftest.py - Added comprehensive documentation in NFT README.md explaining final cluster state after NFT execution, including behavior for both DELETE_VOLUME=true and DELETE_VOLUME=false scenarios - Updated module docstring to reflect removal of TEL_NFT_019 Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>" * Remove TEL_NFT_019 test and document final cluster state after NFT - Removed test_nft_final_cluster_state_warning (TEL_NFT_019) from test_idempotency.py as it is not required as part of NFT suite - Removed corresponding test case entry from conftest.py - Added documentation in NFT README.md explaining final cluster state after NFT execution, including behavior for both DELETE_VOLUME=true and DELETE_VOLUME=false scenarios - Updated module docstring to reflect removal of TEL_NFT_019 Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Implement consolidated NFT cleanup test execution with parametrization - Added @pytest.mark.parametrize to cleanup tests to run twice: Phase 1: cleanup_volume_mode=False (PVCs preserved) Phase 2: cleanup_volume_mode=True (PVCs deleted) - Modified tests in test_performance.py: TEL_NFT_003 (test_cleanup_performance) now parametrized - Modified tests in test_idempotency.py: TEL_NFT_005 (test_cleanup_idempotency) now parametrized TEL_NFT_016/017 (test_cleanup_idempotency_no_pvcs) now parametrized - Added cleanup_volume_modes fixture in conftest.py for future extensibility - Single ./run_validation.sh nft_telemetry test now automatically runs: 1. All existing tests (performance, idempotency, resilience) with PVC preservation 2. Cleanup tests again with PVC deletion enabled - Eliminates need to run NFT suite twice with different DELETE_VOLUME flags Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix cleanup test execution order for consolidated NFT phases - Implemented dynamic ordering using pytest request fixture to assign different order numbers to parametrized cleanup tests: Phase 1 (DELETE_VOLUME=false - PVC preservation): - Order 130: test_cleanup_performance[preserve_pvcs] - Order 131: test_cleanup_idempotency[preserve_pvcs] - Order 132: test_cleanup_idempotency_no_pods - Order 133: test_cleanup_idempotency_no_pvcs[preserve_pvcs] Phase 2 (DELETE_VOLUME=true - PVC deletion): - Order 140: test_cleanup_performance[delete_pvcs] - Order 141: test_cleanup_idempotency[delete_pvcs] - Order 143: test_cleanup_idempotency_no_pvcs[delete_pvcs] - This ensures cleanup tests run in correct logical sequence: 1. Run cleanup without volume 2. Verify results for without-volume scenario 3. Run cleanup with volume 4. Verify results for with-volume scenario - Moved dynamic ordering logic after docstring for proper formatting Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix nft test execution order Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Increase threshold for nft test cases Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> --------- Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>"

…dling (#5356) * Add cleanup_credentials and cleanup_logs preservation flags to telemetry cleanup flow - Add cleanup_credentials and cleanup_logs variables to preserve credentials and logs during cleanup without volume deletion - Add 30-second warning when Delete_sinks_volume=true is combined with preservation flags, as credentials/logs are always deleted in volume mode - Support cleanup_credentials=false for both full cleanup (--tags cleanup) and granular cleanup (--tags cleanup_idrac, etc.) - Modify TEL_NFT_013 test_full_lifecycle to use cleanup_credentials=false for credential preservation during cleanup-then-redeploy cycle - Update cleanup resource summary to display credential and log preservation status - Update README.md files with new flag documentation and usage examples Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Modify Readme for telemetry nft - Remove test_nft_final_cluster_state_warning (TEL_NFT_019) from test_idempotency.py as it is not required as part of NFT suite - Removed corresponding test case entry from conftest.py - Added comprehensive documentation in NFT README.md explaining final cluster state after NFT execution, including behavior for both DELETE_VOLUME=true and DELETE_VOLUME=false scenarios - Updated module docstring to reflect removal of TEL_NFT_019 Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>" * Remove TEL_NFT_019 test and document final cluster state after NFT - Removed test_nft_final_cluster_state_warning (TEL_NFT_019) from test_idempotency.py as it is not required as part of NFT suite - Removed corresponding test case entry from conftest.py - Added documentation in NFT README.md explaining final cluster state after NFT execution, including behavior for both DELETE_VOLUME=true and DELETE_VOLUME=false scenarios - Updated module docstring to reflect removal of TEL_NFT_019 Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Implement consolidated NFT cleanup test execution with parametrization - Added @pytest.mark.parametrize to cleanup tests to run twice: Phase 1: cleanup_volume_mode=False (PVCs preserved) Phase 2: cleanup_volume_mode=True (PVCs deleted) - Modified tests in test_performance.py: TEL_NFT_003 (test_cleanup_performance) now parametrized - Modified tests in test_idempotency.py: TEL_NFT_005 (test_cleanup_idempotency) now parametrized TEL_NFT_016/017 (test_cleanup_idempotency_no_pvcs) now parametrized - Added cleanup_volume_modes fixture in conftest.py for future extensibility - Single ./run_validation.sh nft_telemetry test now automatically runs: 1. All existing tests (performance, idempotency, resilience) with PVC preservation 2. Cleanup tests again with PVC deletion enabled - Eliminates need to run NFT suite twice with different DELETE_VOLUME flags Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix cleanup test execution order for consolidated NFT phases - Implemented dynamic ordering using pytest request fixture to assign different order numbers to parametrized cleanup tests: Phase 1 (DELETE_VOLUME=false - PVC preservation): - Order 130: test_cleanup_performance[preserve_pvcs] - Order 131: test_cleanup_idempotency[preserve_pvcs] - Order 132: test_cleanup_idempotency_no_pods - Order 133: test_cleanup_idempotency_no_pvcs[preserve_pvcs] Phase 2 (DELETE_VOLUME=true - PVC deletion): - Order 140: test_cleanup_performance[delete_pvcs] - Order 141: test_cleanup_idempotency[delete_pvcs] - Order 143: test_cleanup_idempotency_no_pvcs[delete_pvcs] - This ensures cleanup tests run in correct logical sequence: 1. Run cleanup without volume 2. Verify results for without-volume scenario 3. Run cleanup with volume 4. Verify results for with-volume scenario - Moved dynamic ordering logic after docstring for proper formatting Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix nft test execution order Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Increase threshold for nft test cases Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * feat(telemetry): Add cleanup preservation tests and consolidate NFT execution This commit includes multiple improvements to telemetry testing: ## 1. Performance Threshold Updates - VALIDATE_THRESHOLD: 30s -> 45s (50% buffer for config validation) - DEPLOY_THRESHOLD: 600s -> 800s (13.3 min for full stack with all 6 sources) - CLEANUP_THRESHOLD: 300s -> 360s (6 min with 20% infrastructure buffer) - Updated docstrings with detailed threshold calculations ## 2. NFT Test Consolidation - Replaced parametrized cleanup tests with separate functions for correct execution order - Phase 1 (orders 130-133): Cleanup without volume deletion (PVCs preserved) - Phase 2 (orders 140-143): Cleanup with volume deletion (all PVCs deleted) - Single ./run_validation.sh nft_telemetry test now covers both scenarios - Added new TC IDs: TEL_NFT_020, TEL_NFT_021, TEL_NFT_022 - Removed TEL_NFT_019 (moved to README as documentation) ## 3. FVT Cleanup Preservation Tests (NEW) - Added 4 new test cases for cleanup_credentials and cleanup_logs flags: * TEL_FVT_CLEANUP_V015: Verify credentials preserved * TEL_FVT_CLEANUP_V016: Verify credentials deleted * TEL_FVT_CLEANUP_V017: Verify logs preserved * TEL_FVT_CLEANUP_V018: Verify logs deleted - Added helper functions to cleanup_func.py: * verify_credentials_preserved() * verify_credentials_deleted() * verify_logs_preserved() * verify_logs_deleted() - Created new test directory: fvt/cleanup/preservation/ - Updated domain_vars.py to include preservation suite - Total FVT cleanup tests: 18 (14 original + 4 new) ## 4. Documentation Updates - Updated nft/README.md with new test registry and execution order - Updated test/telemetry/README.md with preservation test information - Added FVT_CLEANUP_TEST_FLOW.md with comprehensive test flow documentation - Updated data loss warnings for NFT execution Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Fix environment variable imports in cleanup preservation tests The verification functions were trying to import OMNIA_DATA_PATH and OMNIA_PROJECT_NAME directly from common_vars, but these are environment variable names, not actual values. Changed to: - Import ENV_OMNIA_DATA_PATH and ENV_OMNIA_PROJECT_NAME (the env var names) - Get actual values from host.environment at runtime - Use defaults if environment variables are not set This fixes the ImportError in all 4 preservation test functions: - verify_credentials_preserved() - verify_credentials_deleted() - verify_logs_preserved() - verify_logs_deleted() Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Fix environment variable access in preservation tests The host.environment object doesn't have a .get() method with default parameter. Changed from: host.environment.get(ENV_VAR, default_value) To: host.environment.get(ENV_VAR) or default_value This uses Python's 'or' operator to provide defaults when environment variables are not set, which works correctly with testinfra's Environment object. Fixes AttributeError in all 4 preservation test functions. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Use read_remote_env for host environment variable access Replaced host.environment.get() calls with read_remote_env() from omnia_auto, which is the correct pattern used throughout the codebase. read_remote_env sources /etc/omnia/omnia.env on the target host before reading the variable, ensuring variables set by setup scripts are available in non-login SSH sessions. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Reorder FVT cleanup tests to run preservation first Changed test execution order so preservation cleanup runs FIRST, followed by its verification tests, THEN default cleanup and its verification tests. New execution order: - Order 0: Preservation cleanup playbook (E002) - Order 1-4: Preservation verification tests (V015-V018) - Order 5: Default cleanup playbook (E001) - Order 6-7: Initial cleanup verification (V001-V002) - Order 56-61: Source cleanup verification (V006-V011) - Order 63-65: Sink cleanup verification (V003-V005) - Order 66-67: Final cleanup verification (V012-V014) This ensures: 1. Credentials and logs are preserved first 2. Preservation is verified immediately after 3. Default cleanup runs after preservation is verified 4. Default cleanup verification follows Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Skip preservation tests when delete_sinks_volume=true Added delete_sinks_volume fixture parameter to all preservation tests: - test_deploy_cleanup_with_preservation_flags (E002) - test_cleanup_credentials_preserved (V015) - test_cleanup_credentials_deleted (V016) - test_cleanup_logs_preserved (V017) - test_cleanup_logs_deleted (V018) When delete_sinks_volume=true (cleanup with volume deletion): - Preservation cleanup playbook is skipped - All preservation verification tests are skipped This ensures: 1. Preservation tests only run when PVCs are preserved (default behavior) 2. Preservation tests don't fail when running with DELETE_VOLUME=true 3. Correct test flow: preservation cleanup → preservation verification → default cleanup Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Move preservation tests to cleanup directory for correct ordering pytest-order does not sort correctly across subdirectories. Moved test_cleanup_preservation.py from fvt/cleanup/preservation/ to fvt/cleanup/ so it lives alongside test_playbook.py, ensuring pytest-order can correctly interleave the execution order: Order 0: Preservation cleanup (E002) Order 1: Credentials preserved (V015) Order 2: Credentials deleted (V016) Order 3: Logs preserved (V017) Order 4: Logs deleted (V018) Order 5: Default cleanup (E001) Order 6+: Default cleanup verification (V001-V014) Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Skip default cleanup tests when preservation cleanup runs Added delete_sinks_volume fixture parameter to all default cleanup tests and added skip conditions when delete_sinks_volume=false (preservation cleanup already ran). Tests updated: - test_deploy_cleanup (E001): Skip when delete_sinks_volume=false - test_cleanup_pods_removed (V001): Skip when delete_sinks_volume=false - test_cleanup_topics_removed (V002): Already had skip condition - test_cleanup_idrac (V006): Skip when delete_sinks_volume=false - test_cleanup_ldms (V007): Skip when delete_sinks_volume=false - test_cleanup_ome (V008): Skip when delete_sinks_volume=false - test_cleanup_ufm (V009): Skip when delete_sinks_volume=false - test_cleanup_vast (V010): Skip when delete_sinks_volume=false - test_cleanup_sfm (V011): Skip when delete_sinks_volume=false - test_cleanup_kafka (V003): Skip when delete_sinks_volume=false - test_cleanup_victoria_metrics (V004): Skip when delete_sinks_volume=false - test_cleanup_victoria_logs (V005): Skip when delete_sinks_volume=false - test_no_pods_after_full_cleanup (V012): Skip when delete_sinks_volume=false - test_no_pvcs_after_full_cleanup (V013/V014): Skip when delete_sinks_volume=false Correct execution flow: - delete_sinks_volume=false (default): Order 0-4: Preservation cleanup + verification Order 5+: SKIPPED (default cleanup tests) - delete_sinks_volume=true: Order 0-4: SKIPPED (preservation tests) Order 5+: Default cleanup + verification Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Remove skip conditions and run all cleanup tests sequentially Removed all skip conditions from cleanup tests to run them sequentially: 1. Preservation cleanup phase (orders 0-4): - E002: Deploy cleanup with preservation flags - V015: Verify credentials preserved - V016: Verify credentials deleted - V017: Verify logs preserved - V018: Verify logs deleted 2. Default cleanup phase (orders 5+): - E001: Deploy cleanup (default: delete credentials and logs) - V001-V014: All default cleanup verification tests Removed delete_sinks_volume fixture parameters from: - test_deploy_cleanup_with_preservation_flags - test_cleanup_credentials_preserved - test_cleanup_credentials_deleted - test_cleanup_logs_preserved - test_cleanup_logs_deleted - test_deploy_cleanup - test_cleanup_pods_removed - test_cleanup_topics_removed - All source cleanup tests (V006-V011) - All sink cleanup tests (V003-V005) - All final cleanup tests (V012-V014) Execution flow: 1. Run preservation cleanup and verify credentials/logs are preserved 2. Run default cleanup and verify all resources are deleted 3. All tests run in order without skipping Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Fix syntax errors from sed command The sed command used to remove skip conditions left stray ')' characters and incomplete docstrings. Fixed by: 1. Removing stray ')' characters from function bodies 2. Removing outdated 'Skipped when...' docstring lines 3. Cleaning up empty lines All test files now have valid Python syntax. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Rewrite FVT cleanup tests with correct variable handling Rewrote all FVT cleanup test files to fix syntax errors and correctly handle delete_sinks_volume, cleanup_credentials, and cleanup_logs variables based on the Ansible source code. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Reorder cleanup tests so deletion checks run after default cleanup Reordered FVT cleanup tests so deleted tests run AFTER default cleanup: Order 0: E002 - Preservation cleanup (cleanup_credentials=false, cleanup_logs=false) Order 1: V015 - Verify credentials preserved Order 2: V017 - Verify logs preserved Order 3: E001 - Default cleanup (deletes credentials and logs) Order 4: V016 - Verify credentials deleted Order 5: V018 - Verify logs deleted Order 6: V001 - Verify pods removed Order 7: V002 - Verify topics removed Order 56-67: V003-V014 - Remaining verification tests V016 and V018 now assert actual deletion instead of being informational, since they run after the default cleanup has removed credentials and logs. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix cleanup fvt flow Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix cleanup fvt flow Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * fix(telemetry): Delete test_playbook.py (moved to test_cleanup_preservation.py) The test_playbook.py file became a documentation-only file after moving the default cleanup deployment into test_cleanup_preservation.py to ensure proper execution order. This caused a pylint score of 0 (below threshold). Since all cleanup deployments are now in test_cleanup_preservation.py and the file serves no purpose, it has been deleted. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update Readme files in fvt Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * refactor(telemetry): Rename test_cleanup_preservation.py to test_playbook.py Renamed the file to better reflect its purpose: it contains both cleanup playbook deployments (E001, E002) and their dependent verification tests (V015-V018), making it the central playbook execution module for cleanup tests. Updated docstring to remove reference to 'in test_playbook.py' since all tests are now in this single file. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> --------- Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>"

…loyment (#5362) * feat(telemetry): support non-destructive iDRAC disable Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * feat(telemetry): add selective sink deployment Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * test(telemetry): cover bridge-derived sink enablement Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> --------- Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>

…isable (#5358) * Issue 4849: Add PowerScale metrics and logs independent enable/disable This commit implements the ability to independently enable/disable PowerScale metrics and logs collection through telemetry_config.yml flags. Changes: - Updated enable.yml to add conditional deployment based on metrics_enabled and logs_enabled flags - Added logic to scale down metrics components when transitioning to logs-only mode (metrics_enabled: false, logs_enabled: true) - Added logic to manage vlagent-config ConfigMap for PowerScale syslog configuration when logs_enabled changes - Rewrote disable.yml with comprehensive disable logic that scales down all PowerScale-owned components while preserving shared infrastructure - Fixed ansible-lint by replacing shell with command for vlagent-config patches Supported combinations: - metrics_enabled: true, logs_enabled: true (baseline - both enabled) - metrics_enabled: true, logs_enabled: false (metrics only) - metrics_enabled: false, logs_enabled: true (logs only) - metrics_enabled: false, logs_enabled: false (both disabled) Key design principles: - Preserves working true/true baseline (no breaking changes) - Never touches shared infrastructure (VictoriaMetrics, VictoriaLogs, Kafka) - Preserves PVCs, secrets, and Helm state for re-enablement - Idempotent operations - Clear separation of concerns (PowerScale role manages metrics, deploy_victoria role manages logs configuration) This closes issue #4849 Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Address PR review comments from priti-parate - Remove logs-related logic from enable.yml (logs managed via PowerScale API) - Remove scale down logic from enable.yml (scaling only in disable.yml) - Add explanation for VMServiceScrape deletion (prevents vmagent error logs) - Update documentation to reflect logs are managed via PowerScale API - Remove logs_enabled check from main.yml (only metrics_enabled matters) - Replace hardcoded deployment names with variables in disable.yml Changes: - enable.yml: Removed vlagent-config management and scale down operations - enable.yml: Removed logs_enabled checks and variables - disable.yml: Added comment explaining VMServiceScrape deletion necessity - disable.yml: Replaced hardcoded deployment names with variables - disable.yml: Updated documentation for logs configuration - main.yml: Removed logs_enabled from powerscale_enabled calculation - vars/main.yml: Added karavi_observability deployment name variables Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Address additional PR review comments from priti-parate - Use powerscale_enabled from main.yml instead of setting ps_metrics_enabled in enable.yml - Add logic to scale up existing deployments if they are at 0 replicas (handles re-enablement) - Add check to only do fresh Helm deployment if deployments don't exist - Clarify VMServiceScrape deletion comment in disable.yml - Update karavi_observability variable comment to reflect both scale up and down usage Changes: - enable.yml: Removed ps_metrics_enabled fact, use powerscale_enabled from main.yml - enable.yml: Added scale up logic for deployments at 0 replicas - enable.yml: Added check to only do fresh Helm deployment if deployments don't exist - disable.yml: Clarified VMServiceScrape deletion comment - vars/main.yml: Updated karavi_observability variable comment Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Fix ansible-lint indentation error in enable.yml Fix indentation for 'when' condition in Display scale up status task. Expected 6 spaces but found 4 spaces. Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Remove VMServiceScrape deletion from disable.yml per reviewer suggestion Following reviewer's suggestion to not delete VMServiceScrape during disable. VMServiceScrape is created by Helm chart during initial deployment and will persist during disable. vmagent may generate errors when scraping scaled-down services, but this is acceptable for operational simplicity. This approach avoids the complexity of managing VMServiceScrape lifecycle while accepting some error logs in vmagent as a trade-off. Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Fix gaps in disable.yml implementation - Remove outdated comment about VMServiceScrape deletion (removed in previous commit) - Replace hardcoded 'csi-volume-exporter' with variable {{ csi_volume_exporter.app_name }} - Ensure consistency with enable.yml which uses variables for deployment names Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> --------- Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> Co-authored-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com>

* Add PowerScale test automation for metrics channel transitions - Add is_metrics_enabled() helper function to telemetry_func.py - Fix powerscale_func.py to read metrics_enabled from telemetry_sources.powerscale - Fix powerscale_func.py to read legacy config from powerscale_configurations - Add skip check for metrics_enabled in test_powerscale.py FVT test - Add NFT test for PowerScale metrics channel transitions Test Coverage: - Metrics enable/disable transitions - Idempotency verification - Cleanup validation Note: Logs are managed via PowerScale API, not by Omnia, so only metrics transitions are tested in automation. Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Fix NFT test to set logs_enabled=false for metrics transitions The source code implementation now uses OR logic for PowerScale enablement: powerscale_enabled = metrics_enabled OR logs_enabled This means if the base config has logs_enabled=true, setting metrics_enabled=false would not disable PowerScale. The NFT test now explicitly sets logs_enabled=false in all test steps to ensure metrics transitions are tested independently. Changes: - Add logs_enabled=False to extra_vars in _deploy_with_channels - Update test documentation to reflect logs_disabled in all steps - Add note about logs_enabled being explicitly set to False Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> --------- Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> Co-authored-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com>

…ry sinks (#5368) * Add dependency-aware selective cleanup for sinks Add cleanup_sinks tag with runtime dependency checking for selective Kafka/VictoriaMetrics/VictoriaLogs cleanup. Supports both -e sinks=kafka,victoria_metrics and short-form -e kafka -e victoria_logs. Includes dependency detection, volume protection, and fail-safe handling. Removed deprecated individual sink tags. Added FVT tests. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update cleanup_sinks.yml Add cleanup_sinks tag with runtime dependency checking for selective Kafka/VictoriaMetrics/VictoriaLogs cleanup. Supports both -e sinks=kafka,victoria_metrics and short-form -e kafka -e victoria_logs. Includes dependency detection, volume protection, and fail-safe handling. Removed deprecated individual sink tags. Added FVT tests. Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update initialize.yml Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update initialize.yml Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update initialize.yml Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Fix cleanup sinks logic * Fix test automation for cleanup_sinks * Changes in telemetry for cleanup_sinks * Fix telemetry cleanup test cases * Fix telemetry cleanup test cases * Fix ansible and pylint issues Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update omnia.sh * Fix ansible lint issues Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Remove stale files Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> --------- Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>

…e Kafka topics (#5375) * feat(telemetry): reconcile OME enable disable lifecycle * docs(telemetry): clarify deploy_sinks -e sinks= usage Update deploy_sinks.yml and telemetry.yml header comments to document the correct extra-var format for direct ansible-playbook invocations: -e sinks=kafka -e sinks=kafka,victoria_metrics -e 'sinks=["kafka","victoria_logs"]' Bare -e kafka (without sinks= prefix) is only supported via omnia.sh, which normalises it to sinks=["kafka"] before invoking ansible-playbook. Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * fix(telemetry): gate Kafka topics by enabled sources Remove stale local topic manifests for disabled iDRAC and LDMS sources. Apply and wait for topic manifests only when the corresponding source has metrics enabled, preventing disabled-source topics from being created during OME-only deployments. Add unit coverage for cleanup, application, and readiness gating. Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> --------- Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>

Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>

…nt PowerScale deployment (#5428) Updates TEL_FVT_DEPLOY_V038 to distinguish continuously available state/resource metrics from event-conditioned metrics. Event-conditioned series are only required when matching Kubernetes events exist. The test is skipped when the CSI volume exporter is intentionally not deployed because external-health-monitor-controller is unavailable. Signed-off-by: U, Sushmitha <Sushmitha_U@Dell.com> Co-authored-by: U, Sushmitha <Sushmitha_U@Dell.com>

…allow (#5430) Updates TEL_FVT_DEPLOY_V038 to distinguish continuously available state/resource metrics from event-conditioned metrics. Event-conditioned series are only required when matching Kubernetes events exist. The test is skipped when the CSI volume exporter is intentionally not deployed because external-health-monitor-controller is unavailable. Also adds gitleaks:allow comment for legitimate password variable usage in trigger_powerscale_test_event(). Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>

…5445) * fix(powerscale): TEL_FVT_DEPLOY_V038 false-negative fix and gitleaks allow Updates TEL_FVT_DEPLOY_V038 to distinguish continuously available state/resource metrics from event-conditioned metrics. Event-conditioned series are only required when matching Kubernetes events exist. The test is skipped when the CSI volume exporter is intentionally not deployed because external-health-monitor-controller is unavailable. Also adds gitleaks:allow comment for legitimate password variable usage in trigger_powerscale_test_event(). Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> * Update deploy.yml Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> * Update deploy_sinks.yml Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> * Update test_ome_lifecycle.py Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> --------- Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>

…stencies (#5441) * Fix PowerScale disable logic and status reporting inconsistencies PowerScale Fixes: 1. Config Caching: Force config reload in deploy_powerscale main.yml 2. deploy.yml Condition: Check values file path instead of metrics_enabled 3. Cert-manager Scaling: Scale down in disable.yml, scale up in enable.yml 4. Status Reporting: Check actual deployment state (replicas, pods) for accurate status 5. Remove duplicate status setting from deploy.yml (overwrites role's status) 6. Fix namespace variable in update_status.yml (use csm_namespace) 7. Fix display scale up status to only check main components (karavi, otel, csi) 8. Fix pod label in status check (karavi-metrics-powerscale instead of karavi-observability) 9. Add 180s wait for pods to be Running before reporting status 10. Add vlagent pod check for logs status PowerScale Status Logic (matches LDMS pattern): - deployed: metrics_enabled=true AND deployment exists AND replicas>0 AND pods running - disabled: metrics_enabled=false AND deployment exists AND replicas=0 - failed: metrics_enabled=true BUT (deployment doesn't exist OR replicas=0 OR pods not running) - skipped: metrics_enabled=false AND deployment doesn't exist PowerScale Logs Status Logic: - deployed: logs_enabled=true AND vlagent pods running - disabled: logs_enabled=false AND PowerScale was previously deployed - failed: logs_enabled=true BUT vlagent pods not running - skipped: logs_enabled=false AND PowerScale never deployed OME Status Fix: - Changed OME status from 'disabled' to 'skipped' when not enabled - Makes OME consistent with UFM, VAST, and LDMS behavior - All sources now show 'skipped' when not configured/enabled vector_ome Bridge Status Fix: - Changed from 'disabled' to 'skipped' when bridge not enabled - Makes vector_ome consistent with vector_ldms behavior PowerScale Expected Flow: 1. Initial (metrics=true): Deploy all 6 components → status='deployed' 2. Change to false: Scale down all 6 to 0 → status='disabled' 3. Change to true: Scale up all 6 to 1 → status='deployed' Components to Scale: - karavi-metrics-powerscale (metrics exporter) - otel-collector (metrics processor) - csi-volume-exporter (PV metrics) - karavi-observability-cert-manager (TLS certificates) - karavi-observability-cert-manager-cainjector (CA injector) - karavi-observability-cert-manager-webhook (certificate webhook) Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Fix PowerScale disable logic and status reporting inconsistencies PowerScale Fixes: 1. Config Caching: Force config reload in deploy_powerscale main.yml 2. deploy.yml Condition: Check values file path instead of metrics_enabled 3. Cert-manager Scaling: Scale down in disable.yml, scale up in enable.yml 4. Status Reporting: Check actual deployment state (replicas, pods) for accurate status 5. Remove duplicate status setting from deploy.yml (overwrites role's status) 6. Fix namespace variable in update_status.yml (use csm_namespace) 7. Fix display scale up status to only check main components (karavi, otel, csi) 8. Fix pod label in status check (karavi-metrics-powerscale instead of karavi-observability) 9. Add 180s wait for pods to be Running before reporting status 10. Add vlagent pod check for logs status PowerScale Status Logic (matches LDMS pattern): - deployed: metrics_enabled=true AND deployment exists AND replicas>0 AND pods running - disabled: metrics_enabled=false AND deployment exists AND replicas=0 - failed: metrics_enabled=true BUT (deployment doesn't exist OR replicas=0 OR pods not running) - skipped: metrics_enabled=false AND deployment doesn't exist PowerScale Logs Status Logic: - deployed: logs_enabled=true AND vlagent pods running - disabled: logs_enabled=false AND PowerScale was previously deployed - failed: logs_enabled=true BUT vlagent pods not running - skipped: logs_enabled=false AND PowerScale never deployed OME Status Fix: - Changed OME status from 'disabled' to 'skipped' when not enabled - Makes OME consistent with UFM, VAST, and LDMS behavior - All sources now show 'skipped' when not configured/enabled vector_ome Bridge Status Fix: - Changed from 'disabled' to 'skipped' when bridge not enabled - Makes vector_ome consistent with vector_ldms behavior PowerScale Expected Flow: 1. Initial (metrics=true): Deploy all 6 components → status='deployed' 2. Change to false: Scale down all 6 to 0 → status='disabled' 3. Change to true: Scale up all 6 to 1 → status='deployed' Components to Scale: - karavi-metrics-powerscale (metrics exporter) - otel-collector (metrics processor) - csi-volume-exporter (PV metrics) - karavi-observability-cert-manager (TLS certificates) - karavi-observability-cert-manager-cainjector (CA injector) - karavi-observability-cert-manager-webhook (certificate webhook) Automation Test Updates: - Removed cleanup_powerscale function and cleanup_powerscale tag usage - Added telemetry status verification from output file - Updated test to cover all 6 transitions (metrics/logs combinations) - Fixed TestLogger method calls (failed instead of error/fail) - Fixed status file parsing (read YAML structure correctly) - Changed to directly modify telemetry_config.yml instead of using extra_vars - Removed custom marker (use standard markers) Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> * Remove redundant when condition from PowerScale deploy import Addresses review feedback: the deploy_powerscale role already decides enable/disable from telemetry_config, so the import-level condition is not required. Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> --------- Signed-off-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com> Co-authored-by: sushmitha-dell <sushmitha-dell@users.noreply.github.com>

…st fixes (#5467) * feat(repo_manager): support multi-version RHEL artifact reconciliation (#5348) Add sequential RHEL minor-version contexts, version-qualified status endpoints, verified shared artifact reuse, container platform validation, and RPM repository recovery checkpoints.\n\nPreserve existing public tags, logs, status tables, and Pulp naming while documenting rerun, resync, cleanup, and policy behavior. * feat(build_stream): implement cadence polling and unified pipeline (ER-BSM-002-cadence-polling-build) (#5345) (#5349) * Implement cadence polling for BuildStream * Fix CI failures * Address review comments --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> * Add multi-version RHEL support with per-version base packages and python3 fix (#5353) * Add per-version base image package support and improve cleanup - Add base_image_packages_by_version to support different package lists for RHEL 10.0 and 10.2 base images - Update parse_catalog.py to handle version-specific base packages - Change base image builds from sequential to parallel (max_parallel) - Update build templates to use version-specific package lists - Make cleanup_build_output version-aware to preserve other RHEL versions - Add cleanup_build_output_scope parameter (current/all) - Default: current (safe, preserves other versions) - all: destructive, removes all outputs (requires explicit opt-in) - Add cleanup_hash_cache.yml to remove package hash cache files - Update fetch_repo_manager_repos.yml for improved repo handling - Remove version pin checks from base image builds (orchestrator handles failures) This enables building base images for multiple RHEL versions (10.0, 10.2) with appropriate package lists per version, and allows selective cleanup without destroying historical build outputs. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add RHEL 10.0 and 10.2 catalog with python3 version fix - Add catalog_rhel_10_0_rhel_10_2_x86.json for multi-version RHEL support - Remove version pin from python3 packages to allow dnf to resolve correct version per RHEL release (3.12.9 for 10.0, 3.12.14 for 10.2) - Update python3_3_12_9 to use unversioned python3 package - Update service_k8s_cluster_group and service_kube_control_plane_group to use python3 instead of python3_3_12_9 This fixes the python3-devel version conflict where python3-devel-3.12.14 required python3 >= 3.12.14 but the catalog pinned python3-3.12.9 for both RHEL 10.0 and 10.2. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Simplify cleanup tags - use --tags cleanup without special tags - Remove individual tags (minio, registry, output, s3cmd, etc.) from cleanup tasks - All cleanup tasks now run with single --tags cleanup - Simplify usage: ansible-playbook image_build_manager.yml --tags cleanup - Update documentation to reflect simplified tag usage - Keep cleanup_build_output_scope parameter for selective cleanup This makes cleanup simpler and more intuitive without needing to remember multiple tag names. Users can still selectively clean with: --tags cleanup -e cleanup_build_output_scope=all Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Simplify cleanup - remove all build outputs by default - Remove cleanup_build_output_scope parameter - Cleanup now deletes entire output_project_dir by default - Simplify cleanup_build_output.yml to remove all status files - Since images are deleted, there's no need to preserve status files - Cleanup is now straightforward: --tags cleanup removes everything Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Update main.yml Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Delete src/main/samples/catalog_rhel_10_0_rhel_10_2_x86.json Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> --------- Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * feat(repo_manager): support secure RHEL 10.0 and 10.2 reconciliation (#5366) * feat(repo_manager): support multi-version RHEL artifact reconciliation Add sequential RHEL minor-version contexts, version-qualified status endpoints, verified shared artifact reuse, container platform validation, and RPM repository recovery checkpoints.\n\nPreserve existing public tags, logs, status tables, and Pulp naming while documenting rerun, resync, cleanup, and policy behavior. * feat(repo_manager): support secure RHEL 10.0 and 10.2 reconciliation Signed-off-by: Narthan_S <narthan.s@dell.com> * Delete src/repo_manager/scripts directory Signed-off-by: snarthan <narthan.s@dell.com> --------- Signed-off-by: Narthan_S <narthan.s@dell.com> Signed-off-by: snarthan <narthan.s@dell.com> * Sync pub/build_stream → pub/rhel10.2 (Bidirectional Phase 1) (#5382) * feat(build_stream): implement cadence polling and unified pipeline (ER-BSM-002-cadence-polling-build) (#5345) * Implement cadence polling for BuildStream Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Fix CI failures Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Address review comments Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Merge pub/rhel10.2 to pub/build_stream (#5352) * feat(repo_manager): support multi-version RHEL artifact reconciliation (#5348) Add sequential RHEL minor-version contexts, version-qualified status endpoints, verified shared artifact reuse, container platform validation, and RPM repository recovery checkpoints.\n\nPreserve existing public tags, logs, status tables, and Pulp naming while documenting rerun, resync, cleanup, and policy behavior. * feat(image_build_manager): add global image dictionary (#5334) * feat(image_build_manager): add global image dictionary Implement dictionary-aware catalog image reuse, catalog-versioned artifacts and status contracts, exact cleanup pruning, and BuildStream differential or lockstep execution integration. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(image_build_manager): use one rebuild control Use build_image.force_rebuild as the single rebuild control for direct and BuildStream catalog builds, and align catalog schema version handling to an integer. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * test(image_build_manager): cover catalog dictionary workflows Add isolated catalog reuse and rebuild scenarios, focused dictionary unit coverage, and enforce the latest project-level plus catalog-versioned build-status contracts. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): remove duplicate rebuild mode Use image_build_manager build_image.force_rebuild as the only rebuild control and remove the obsolete differential/lockstep configuration surface and tests. Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * docs(image_build_manager): document catalog reuse commands Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix: address global dictionary review comments Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> Co-authored-by: snarthan <narthan.s@dell.com> Co-authored-by: SOWJANYAJAGADISH123 <Sowjanya.Jagadish@dell.com> * refactor(build_stream): remove deprecated catalog_roles and generate_input_files APIs (#5354) Remove deprecated APIs and infrastructure retired in Omnia 2.3 to simplify the architecture and consolidate build stages. Each domain now consumes the catalog directly instead of relying on intermediate API layers. Key Changes: - Remove /api/v1/catalog_roles API (deprecated in Omnia 2.3) - Remove /api/v1/generate_input_files API (deprecated in Omnia 2.3) - Remove GenerateInputFilesUseCase and related commands - Remove nfs_build_image_inventory_repository (SQL repository is active) - Consolidate build-image-x86_64 and build-image-aarch64 stages into unified build-image stage - Simplify image_build_manager.yml entry point for all architectures - Remove architecture-specific validation logic (364 lines) - Clean up dependency injection container configuration - Update documentation to reflect API retirement and stage consolidation - Remove tests for deleted code (13 test files, ~3,600 lines) Impact: - No functional impact - all removed code was deprecated in Omnia 2.3 - Domains (repo_manager, image_build_manager) now consume catalog directly - Simplified architecture with reduced code duplication - Single build-image stage handles all architectures via playbook dispatch Net Change: -5,666 lines (5,825 deletions, 159 additions) Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * feat(ER-BSM-002): complete catalog identity and repository reconciliation (#5355) * fix(build_stream): persist catalog artifact identity Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * feat(repo_manager): add catalog exact-mirror reconciliation Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(repo_manager): harden repository metadata validation Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): skip remote cleanup without GitLab host (#5357) Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * Integration Issues and fixes found during testing (#5371) * refactor(build_stream): remove deprecated catalog_roles and generate_input_files APIs Remove deprecated APIs and infrastructure retired in Omnia 2.3 to simplify the architecture and consolidate build stages. Each domain now consumes the catalog directly instead of relying on intermediate API layers. Key Changes: - Remove /api/v1/catalog_roles API (deprecated in Omnia 2.3) - Remove /api/v1/generate_input_files API (deprecated in Omnia 2.3) - Remove GenerateInputFilesUseCase and related commands - Remove nfs_build_image_inventory_repository (SQL repository is active) - Consolidate build-image-x86_64 and build-image-aarch64 stages into unified build-image stage - Simplify image_build_manager.yml entry point for all architectures - Remove architecture-specific validation logic (364 lines) - Clean up dependency injection container configuration - Update documentation to reflect API retirement and stage consolidation - Remove tests for deleted code (13 test files, ~3,600 lines) Impact: - No functional impact - all removed code was deprecated in Omnia 2.3 - Domains (repo_manager, image_build_manager) now consume catalog directly - Simplified architecture with reduced code duplication - Single build-image stage handles all architectures via playbook dispatch Net Change: -5,666 lines (5,825 deletions, 159 additions) Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Integration Issues and fixes found during testing Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Fix CI issues: add requests dependency and suppress Bandit B501 Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Fix the failing UTs Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> Co-authored-by: snarthan <narthan.s@dell.com> Co-authored-by: SOWJANYAJAGADISH123 <Sowjanya.Jagadish@dell.com> * rhel 10.2 support for slurm benchmarks (#5372) Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> * feat(repo_manager): support secure RHEL 10.0 and 10.2 minor fixed on … (#5389) * feat(repo_manager): support secure RHEL 10.0 and 10.2 minor fixed on the repo_status.yml Signed-off-by: Narthan_S <narthan.s@dell.com> * feat(repo_manager): support secure RHEL 10.0 and 10.2 minor versions fixed on the repo_status.yml Signed-off-by: Narthan_S <narthan.s@dell.com> --------- Signed-off-by: Narthan_S <narthan.s@dell.com> * feat(catalog_skills): add catalog generation, analysis, editing, and diff/changelog skills (#5393) Implements a comprehensive suite of AI skills for catalog management under ER-BSM-001, including master reference file, catalog generation, impact/compatibility analysis, catalog editing with pre-edit gate, and reversible diff/changelog engine. Adds Python tooling for catalog diff/patch operations and fixes schema validation issues. Signed-off-by: venu <236371043+Venu-p1@users.noreply.github.com> * fix(ai-skills): correct impact checks and catalog prompts (#5395) * Fix catalog skill impact analysis, validation, and generation prompts Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> * feat(ai-skills): refine catalog selections and role mappings Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> * feat(ai-skills): add hybrid OS catalog guidance Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> * fix(ai-skills): explicitly offer hybrid OS selection Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> * feat(catalog): add granular diffs and driver-kernel checks Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> * fix(ai-skills): clarify catalog interview and confirmations * feat(ai-skills): review package dependencies before additions --------- Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> * added hybrid catalog files (#5374) * feat: add hybrid catalog files for RHEL 10.2 and 10.0 * feat(repo_manager): support secure RHEL 10.0 and 10.2 reconciliation Signed-off-by: VirajPatil-1500 <VirajPatil-1500@users.noreply.github.com> * removed aarch64 entry for all k8s images * python version issue solved and catalog file path issue --------- Signed-off-by: VirajPatil-1500 <VirajPatil-1500@users.noreply.github.com> Co-authored-by: VirajPatil-1500 <VirajPatil-1500@users.noreply.github.com> * feat(build_stream): implement retention policies and traceability (ER-BSM-002 Story 4) (#5414) * feat(build_stream): implement retention policies and traceability (ER-BSM-002 Story 4) Implements deploy_count tracking, age-based retention, sidecar manifest upload, and audit events for image lifecycle management. Descopes the provenance query API as it duplicates existing data and is not required by BR-86010. Core Changes: - Increment deploy_count and set last_deployed_at on successful deploy (result_poller.py, repositories.py) - Add age-based retention evaluation in cleanup_cron.py with configurable retention_age_days and min_keep_count from build_stream_config.yml - Emit RETENTION_DELETED audit events for aged, undeployed, unprotected images - Add sidecar manifest generation in cadence GitLab CI pipeline (.gitlab-ci-cadence.yml) with catalog_name, catalog_version, git_commit_sha, build_timestamp, job_id Spec Updates: - Descoped FR-7.2 (Provenance Query API) from ER-BSM-002 and Story 4 - Updated spec.md, plan.md, metadata.json, and HLD to reflect descoping - Added descoped-frs field to metadata.json Tests: - Added 14 unit tests for deploy_count increment, retention eligibility, and config loading (test_retention_and_deploy_count.py) - Added 7 FVT tests for retention & traceability verification (fvt/retention_traceability/) - Registered TC IDs BSM_FVT_RETENTION_V001-V007 in test_case_vars.py Validation: - Unit tests: 14 passed, 0 failed, 0 skipped - Overall UTs: 1177 passed, 76 failed (pre-existing), 31 skipped - Pylint: 9.31/10 (product code), 10.00/10 (FVT tests) - No regressions introduced Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * fix(build_stream): implement abstract methods in InMemoryImageGroupRepository Implement increment_deploy_count and list_eligible_for_retention in InMemoryImageGroupRepository to fix unit test failures caused by new abstract methods in ImageGroupRepository interface. Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * chore(build_stream): suppress pylint W2301 in abstract methods Suppress unnecessary-ellipsis warnings in ImageGroupRepository ABC methods as they follow the standard pattern of using ... for abstract method bodies. Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * chore(test): fix pylint issues in retention unit tests Remove unused MagicMock import and split long import line for test_retention_and_deploy_count.py to achieve 10.00/10 pylint score. Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> * Enhance container platform validation with improved error logging and OCI attestation support Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Update repo_manager_config.yml with RHEL 10.2 support Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Fix registry configuration documentation to use correct snake_case format Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Uncomment user_repos sections for all RHEL versions Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Remove example format comments from RHEL subscription repos Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Remove rhel-10-for-x86_64-highavailability-rpms repository entries Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Update user_repos priority from 100 to 10 for RHEL 10.0 Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Update pull_benchmarks.sh.j2 as per hpc benchmarks (#5419) Signed-off-by: Nagachandan P <Nagachandan.p@dell.com> * fix(image_build_manager): align versioned artifacts, catalogs, and registry (#5411) * fix(image_build_manager): record versioned Thrillhouse boot artifacts Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * fix(security): resolve Checkmarx command injection findings Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * fix: align Thrillhouse catalog consumers and validation Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> --------- Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * refactor(image-build): align role structure and dependency checks (#5426) * refactor(image-build): align roles and tests for #5411 Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * fix(deps): align netaddr versions for #5411 Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * fix(test): harden catalog reuse recovery state Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> --------- Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * test(orchestrator): align FVT with lifecycle status Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * test(orchestrator): complete lifecycle automation coverage Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * feat(telemetry, orchestrator, repo_manager): add RHEL 10.2 dynamic OS version support and fix Python distribution URL routing (#5434) * feat(telemetry): add RHEL 10.2 support with dynamic OS version detection - Derive pulp_os_version dynamically from orchestrator inventory group names (e.g. *_rhel_10_2_* -> 10.2) instead of hardcoding '10.0' - Add fail task with clear error message if OS version cannot be determined and repo_url is not explicitly set - Remove hardcoded pulp_os_version: '10.0' from common/vars/main.yml - Fix repo_url resolution in install_python_dependencies to honour telemetry_packages_config resolved value from localhost hostvars - Add mirror entries for registry.connect.redhat.com, registry.redhat.io and registry.access.redhat.com in all three OChaMI metadata service templates to route Red Hat registry pulls through the Pulp mirror - Fix inventory group loop in deploy_idrac_telemetry playbook and initiate_telemetry_service_cluster tasks to dynamically select all groups matching '^service_kube_node' instead of hardcoded 'service_kube_node_x86_64', supporting multi-version inventories - Update comments and example URLs in telemetry_packages.yml, output-contract.md and telemetry_validation.py to reflect dynamic OS version placeholder Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * docs(telemetry): use dynamic os_version placeholder in output-contract Updated output-contract.md to reflect that repo_url is evaluated at runtime based on the orchestrator inventory group names rather than being hardcoded to a specific RHEL version. - Replace hardcoded 'rhel/10.2' with 'rhel/<os_version>' in the example repo_url field - Update packages.repo_url field description to note that OS version is auto-derived from orchestrator inventory group names at runtime Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * fix(security_utils, precheck): accept content route and add version nesting in file_repos - Extend normalize_managed_python_distribution_url to accept both /pypi/<base_path> and /pulp/content/<base_path> routes from the internal Pulp authority, and always return the canonical /pulp/content/ URL for consistent Repo Manager consumers. - Add version nesting level to file_repos in _published_repositories so iteration follows version -> architecture -> repo_type -> name, matching the updated repo_status.yml schema. Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * fix(repo_manager): add PyMySQL cleanup example and renumber test case IDs Add cleanup command example for PyMySQL in repo_manager.yml comments. Renumber ExactMirror test case IDs (144-154 -> 213-223) to resolve conflicts with newly added test cases. Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> --------- Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> * Pub/rhel10.2 correct catalog architecture mappings and Slurm package (#5433) * fix(catalogs): isolate Image Builder containers by architecture and restore OS functional layers Fixes #949 Signed-off-by: VirajPatil-1500 <Viraj.Patil@dellteam.com> * fixes thrillhouse image tag version --------- Signed-off-by: VirajPatil-1500 <Viraj.Patil@dellteam.com> * feat(build_stream): always run cadence and add FVT lifecycle (#5420) * feat(build_stream): add cadence force build validation Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * test(build_stream): extend cadence pipeline sanity coverage Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): always trigger cadence after repository sync Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): align cadence validation contracts Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): align cadence configuration contract Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(ER-BSM-002-cadence-polling-build): harden cadence sync Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): remove residual runtime artifacts during cleanup (#5429) * fix(build_stream): remove residual runtime artifacts during cleanup Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * fix(build_stream): align cleanup watcher lifecycle Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * docs(build_stream): align watcher service name Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * docs(build_stream): document cleanup redeployment flow Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * test(build_stream): fail cleanup inspection errors Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * test(build_stream): isolate cleanup inspection classifiers Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> --------- Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> * refactor(lifecycle): remove unsupported upgrade and rollback flows (#5446) Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * test(orchestrator): install source-compiled MPI on non-OFED nodes before verification On nodes without Mellanox/OFED hardware, DOCA MPI is absent and the source-compiled stack is deployed but never auto-executed. The OpenMPI and UCX verification functions now detect non-OFED nodes, run the install scripts on the login_compiler_node (compiling to NFS), configure env vars on each non-OFED compute node, then re-discover and verify. When no login_compiler_node is present in the cluster, the affected tests skip with a clear reason instead of hard-failing. Changes: - pxeboot_vars.py: add mellanox_detect, source_mpi_check, install_ucx_source, install_openmpi_source, configure_ucx_openmpi_env, script_exists commands - _workload_helpers.py: add ensure_source_mpi() shared helper with idempotent compile-once caching - slurm_pxeboot_func.py: two-pass discovery in check_slurm_openmpi_installation — detect non-OFED failures, install from source, re-verify - slurm_fabric_pxeboot_func.py: same pattern for check_slurm_ucx_transport - slurm_jobs_pxeboot_func.py: same pattern for check_slurm_openmpi_job Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * test(orchestrator): fix Slurm reboot recovery and drain-queue verification check_slurm_cluster_recovery: - Fix KeyError: missing slurm_sbatch command in PXEBOOT_COMMANDS - Fix openmpi_job called without format args (5 %s params required) - Add per-node boot ID tracking (verifies each node actually rebooted) - Add per-node cloud-init phase reporting - Add Slurm state recovery polling (waits for idle after reboot) - Report each phase individually per node instead of aggregate pass/fail - Build specific error messages listing which nodes failed which phase check_slurm_drain_queue_recovery: - Add drain state verification before submitting job (poll sinfo) - Add resume state verification after undrain (poll for idle) - Report initial state, drain state, job state, and resume state - Add specific error messages for each failure phase - Wrap finally-block cleanup in try/except to prevent silent failures New PXEBOOT_COMMANDS: - slurm_sbatch: simple sbatch without user identity - slurm_drain_state: poll sinfo for a node's current state Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(test): add 7 missing PXEBOOT_COMMANDS that cause runtime KeyError kubernetes_deployments: kubectl get deployments -A (JSON) for NFS provisioner container and backend validation mount_contract: findmnt -J (JSON) for per-node NFS mount source verification etcd_setup_script: test -x /usr/local/bin/etcd-disk-setup.sh etcd_update_script: test -x /usr/local/bin/etcd-fstab-update.sh etcd_setup_log: test -s /var/log/etcd-disk-setup.log etcd_update_log: test -s /var/log/diskless-etcd-mount.log etcd_selection_log: cat both etcd logs for disk-selection regex parsing Static validation confirms all 146 referenced command keys now resolve. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Refactor repo_manager FVT framework to align with image_build_manager structure (#5412) * Add build_stream pipeline with prepare_base stage and cleanup_build_stream - Added .gitlab-ci-build-stream.yml with build_stream pipeline support - Added cleanup_build_stream stage to both build-stream and cluster pipelines - Added prepare_base stage to build-stream pipeline (combines repo_manager, image_build_manager, orchestrator setup) - Removed individual deploy stages (repo_manager, image_build_manager, orchestrator) from build-stream - Updated test stages to depend on prepare_base instead of individual deploy stages - Updated summary stage to reflect new pipeline structure - Updated documentation for build_stream pipeline changes - Updated setup_gitlab_project.py for build_stream support Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * fixed variable name Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * updated build_stream pipeline Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add comprehensive build_stream pipeline support to setup scripts and documentation This commit adds full build_stream pipeline support across the Omnia CI/CD pipeline: **setup_gitlab_project.py changes:** - Fixed load_pipeline_config() to read build_stream_enable from pipeline_config.yml - Fixed generate_cluster_build_stream_trigger_job() to include BUILD_STREAM_TAGS variable - Fixed cmd_create() config-file mode to set BUILD_STREAM_ENABLE global default - Ensured all build_stream input files and test configs are collected and committed **pipeline_config.yml changes:** - Added global.build_stream_enable setting to enable build_stream pipeline mode - Added deploy_tags.build_stream for ansible tags configuration - Added test_commands.build_stream with default test command - Updated domains documentation to mention build_stream as valid domain - Updated skip_stages documentation to include build_stream **Documentation additions:** - Created comprehensive BUILD_STREAM_PIPELINE.md guide covering: * Overview and key differences from cluster pipeline * Pipeline stages and conditional execution * prepare_base stage explanation * Configuration reference for all variables * Real-world deployment examples * Input files and credentials setup * Troubleshooting guide - Updated docs/README.md to reference BUILD_STREAM_PIPELINE.md - Added build_stream links to common tasks and getting started sections **Pipeline YAML changes:** - .gitlab-ci-build-stream.yml: Added TEST_BUILD_STREAM_CMD fallback logic - .gitlab-ci-cluster.yml: Added TEST_CMD fallback logic for all test stages - .gitlab-ci-utils.yml: Added TEST_UTILS_CMD fallback logic - .gitlab-ci.yml: Already includes build_stream trigger job generation **Test command fallback mechanism:** All test stages now use bash parameter expansion to fall back to defaults if CI/CD variables are empty or unset, preventing empty command execution. **Report naming:** Summary stages now rename downloaded test reports to include pipeline ID for unique identification across multiple pipeline runs. **Test Execution Summary:** Email notifications now show all domains with appropriate status: - Counts for domains with report data - "No Report" for executed tests without report files - "Not Executed" for test stages that didn't run Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix DRY_RUN report fetching, cleanup_utils status, and old report attachment issues **Issue 1: Test reports being fetched and attached during DRY_RUN** - Added DRY_RUN check before fetching test reports in summary stage - When DRY_RUN=true, report fetching is completely skipped - Prevents unnecessary SSH operations and report attachments during dry runs - Applied to .gitlab-ci-cluster.yml and .gitlab-ci-build-stream.yml **Issue 2: cleanup_utils showing UNKNOWN status despite running** - Added cleanup_utils to summary stage's needs section - Ensures summary stage waits for cleanup_utils to complete before collecting job statuses - Prevents race condition where summary runs before cleanup_utils finishes - cleanup_utils status will now correctly show PASSED/FAILED instead of UNKNOWN - Applied to .gitlab-ci-cluster.yml and .gitlab-ci-build-stream.yml **Issue 3: Old test reports being attached with new pipeline ID prefix** - Removed unsafe Pattern 2 fallback that matched reports without pipeline ID - Now only fetches reports with current pipeline ID to prevent picking up old reports - Pattern 1: Exact match with current pipeline ID - Pattern 2: Broad match with current pipeline ID (was Pattern 3) - This prevents old reports from previous successful runs from being attached when current pipeline fails during initialization - Applied to .gitlab-ci-cluster.yml, .gitlab-ci-build-stream.yml, and .gitlab-ci-utils.yml These fixes ensure: 1. Dry runs don't attach test reports or generate unnecessary artifacts 2. All job statuses are properly captured for email notifications 3. Pipeline execution summary accurately reflects all stage statuses 4. Only reports from the current pipeline are attached to email notifications 5. Old reports from previous runs are never mistakenly attached Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Refactor FVT framework to align with image_build_manager structure - Reorganized FVT tests by playbook tags with test_playbook.py drivers - Split test_status.py files into suite subdirectories (config, pulp, status, cleanup) - Moved catalog tests into separate tag directories (catalog_generate, catalog_add, catalog_delete, catalog_validate) - Moved policy tests into execute/policy/ suite - Distributed negative tests into tag-specific negative/ suites - Removed old catalog/, policy/, and negative/ directories - Updated domain_vars.py with new FVT_TAGS, SUITES, and VERIFY_ONLY_SUITES - Added tl.skipped_fields() to all 51 skip scenarios for professional output - Added pytest_report_teststatus hook to suppress pytest progress characters - Created negative suite under user_registry and moved test_user_registry_negative.py - Updated all __init__.py docstrings and README.md Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * remove the unncessary md files Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix pylint issues in user_registry test_playbook.py - Remove duplicate import of run_playbook - Remove unused import of _get_input_path Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add catalog_validate directory and fix hardcoded catalog path - Create catalog_validate/ directory with validate/ suite - Add test_playbook.py for catalog_validate tag - Add test_validate.py with 3 verification tests: - RM_FVT_CATALOG_VALIDATE_V001: Verify catalog validation completed - RM_FVT_CATALOG_VALIDATE_V002: Verify catalog validation log file exists - RM_FVT_CATALOG_VALIDATE_V003: Verify catalog file still valid after validation - Fix hardcoded catalog path in catalog_generate/generate/test_generate.py - Replace os.environ.get('CATALOG_FILE_PATH', '/opt/omnia/catalog/catalog_rhel.json') - Use _get_catalog_path() function for consistency - Remove unused os import This completes the catalog tag structure and ensures all catalog paths use dynamic configuration functions. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Delete test/repo_manager/update_remaining_skips.py Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Update test_config.yml Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> --------- Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> * fix(orchestrator): resolve timezone and hostvars gaps in metadata-service templates Cloud-init templates used ansible_date_time.tz which returns timezone abbreviations (EDT, EST, CDT) instead of IANA names (America/New_York). timedatectl set-timezone EDT fails silently, leaving /etc/localtime missing and breaking Apptainer container execution. - Add IANA timezone resolution in provision_common before template rendering - Fix 11 metadata-service templates to use 'UTC' fallback instead of broken ansible_date_time.tz - Set oim_timezone fact in validate_oim_timezone.yml for precheck consumers - Fix hostvars['oim'] -> hostvars['localhost'] for variables set by passwordless_ssh role (slurm_ssh_patterns, k8s_ssh_patterns, all_group_names_present, login_compiler_node_present, slurm_node_present) Resolves Apptainer FATAL errors on container execution due to missing /etc/localtime on PXE-booted nodes. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * fix(orchestrator,image_build_manager): migrate to version-qualified repo_status contract and harden aarch64 SSH (#5448) Orchestrator: migrate from deprecated flat offline_*_path variables to the version-qualified base_urls and file_repos maps emitted by Repo Manager. Add catalog context resolution for service Kubernetes OS version and architecture, and normalize the repo_status contract per version/arch slice. Image Build Manager: harden aarch64 SSH key management to prevent 'too many authentication attempts' failures during remote node preparation. Fix load_config to validate build configuration before proceeding. Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> * Merge pull request #5394 from SujalS27/orch_stage test(orchestrator): add Slurm node remove/add lifecycle verification * Feature branch sync - staging to pub/rhel10.0 (#5451) * refactor(lifecycle): remove unsupported upgrade and rollback flows (#5446) Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * Refactor repo_manager FVT framework to align with image_build_manager structure (#5412) * Add build_stream pipeline with prepare_base stage and cleanup_build_stream - Added .gitlab-ci-build-stream.yml with build_stream pipeline support - Added cleanup_build_stream stage to both build-stream and cluster pipelines - Added prepare_base stage to build-stream pipeline (combines repo_manager, image_build_manager, orchestrator setup) - Removed individual deploy stages (repo_manager, image_build_manager, orchestrator) from build-stream - Updated test stages to depend on prepare_base instead of individual deploy stages - Updated summary stage to reflect new pipeline structure - Updated documentation for build_stream pipeline changes - Updated setup_gitlab_project.py for build_stream support Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * fixed variable name Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * updated build_stream pipeline Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add comprehensive build_stream pipeline support to setup scripts and documentation This commit adds full build_stream pipeline support across the Omnia CI/CD pipeline: **setup_gitlab_project.py changes:** - Fixed load_pipeline_config() to read build_stream_enable from pipeline_config.yml - Fixed generate_cluster_build_stream_trigger_job() to include BUILD_STREAM_TAGS variable - Fixed cmd_create() config-file mode to set BUILD_STREAM_ENABLE global default - Ensured all build_stream input files and test configs are collected and committed **pipeline_config.yml changes:** - Added global.build_stream_enable setting to enable build_stream pipeline mode - Added deploy_tags.build_stream for ansible tags configuration - Added test_commands.build_stream with default test command - Updated domains documentation to mention build_stream as valid domain - Updated skip_stages documentation to include build_stream **Documentation additions:** - Created comprehensive BUILD_STREAM_PIPELINE.md guide covering: * Overview and key differences from cluster pipeline * Pipeline stages and conditional execution * prepare_base stage explanation * Configuration reference for all variables * Real-world deployment examples * Input files and credentials setup * Troubleshooting guide - Updated docs/README.md to reference BUILD_STREAM_PIPELINE.md - Added build_stream links to common tasks and getting started sections **Pipeline YAML changes:** - .gitlab-ci-build-stream.yml: Added TEST_BUILD_STREAM_CMD fallback logic - .gitlab-ci-cluster.yml: Added TEST_CMD fallback logic for all test stages - .gitlab-ci-utils.yml: Added TEST_UTILS_CMD fallback logic - .gitlab-ci.yml: Already includes build_stream trigger job generation **Test command fallback mechanism:** All test stages now use bash parameter expansion to fall back to defaults if CI/CD variables are empty or unset, preventing empty command execution. **Report naming:** Summary stages now rename downloaded test reports to include pipeline ID for unique identification across multiple pipeline runs. **Test Execution Summary:** Email notifications now show all domains with appropriate status: - Counts for domains with report data - "No Report" for executed tests without report files - "Not Executed" for test stages that didn't run Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix DRY_RUN report fetching, cleanup_utils status, and old report attachment issues **Issue 1: Test reports being fetched and attached during DRY_RUN** - Added DRY_RUN check before fetching test reports in summary stage - When DRY_RUN=true, report fetching is completely skipped - Prevents unnecessary SSH operations and report attachments during dry runs - Applied to .gitlab-ci-cluster.yml and .gitlab-ci-build-stream.yml **Issue 2: cleanup_utils showing UNKNOWN status despite running** - Added cleanup_utils to summary stage's needs section - Ensures summary stage waits for cleanup_utils to complete before collecting job statuses - Prevents race condition where summary runs before cleanup_utils finishes - cleanup_utils status will now correctly show PASSED/FAILED instead of UNKNOWN - Applied to .gitlab-ci-cluster.yml and .gitlab-ci-build-stream.yml **Issue 3: Old test reports being attached with new pipeline ID prefix** - Removed unsafe Pattern 2 fallback that matched reports without pipeline ID - Now only fetches reports with current pipeline ID to prevent picking up old reports - Pattern 1: Exact match with current pipeline ID - Pattern 2: Broad match with current pipeline ID (was Pattern 3) - This prevents old reports from previous successful runs from being attached when current pipeline fails during initialization - Applied to .gitlab-ci-cluster.yml, .gitlab-ci-build-stream.yml, and .gitlab-ci-utils.yml These fixes ensure: 1. Dry runs don't attach test reports or generate unnecessary artifacts 2. All job statuses are properly captured for email notifications 3. Pipeline execution summary accurately reflects all stage statuses 4. Only reports from the current pipeline are attached to email notifications 5. Old reports from previous runs are never mistakenly attached Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Refactor FVT framework to align with image_build_manager structure - Reorganized FVT tests by playbook tags with test_playbook.py drivers - Split test_status.py files into suite subdirectories (config, pulp, status, cleanup) - Moved catalog tests into separate tag directories (catalog_generate, catalog_add, catalog_delete, catalog_validate) - Moved policy tests into execute/policy/ suite - Distributed negative tests into tag-specific negative/ suites - Removed old catalog/, policy/, and negative/ directories - Updated domain_vars.py with new FVT_TAGS, SUITES, and VERIFY_ONLY_SUITES - Added tl.skipped_fields() to all 51 skip scenarios for professional output - Added pytest_report_teststatus hook to suppress pytest progress characters - Created negative suite under user_registry and moved test_user_registry_negative.py - Updated all __init__.py docstrings and README.md Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * remove the unncessary md files Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix pylint issues in user_registry test_playbook.py - Remove duplicate import of run_playbook - Remove unused import of _get_input_path Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add catalog_validate directory and fix hardcoded catalog path - Create catalog_validate/ directory with validate/ suite - Add test_playbook.py for catalog_validate tag - Add test_validate.py with 3 verification tests: - RM_FVT_CATALOG_VALIDATE_V001: Verify catalog validation completed - RM_FVT_CATALOG_VALIDATE_V002: Verify catalog validation log file exists - RM_FVT_CATALOG_VALIDATE_V003: Verify catalog file still valid after validation - Fix hardcoded catalog path in catalog_generate/generate/test_generate.py - Replace os.environ.get('CATALOG_FILE_PATH', '/opt/omnia/catalog/catalog_rhel.json') - Use _get_catalog_path() function for consistency - Remove unused os import This completes the catalog tag structure and ensures all catalog paths use dynamic configuration functions. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Delete test/repo_manager/update_remaining_skips.py Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Update test_config.yml Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> --------- Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> --------- Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(orchestrator): add DCGM/CUDA FVT coverage and OIM readiness precheck suite (#5443) * test(orchestrator): add HPC benchmarks, CoreDNS, and PowerVault FVT suites for pxeboot lifecycle (#5391) * Add discovery stage support and fix pipeline email reporting (#5454) * Add build_stream pipeline with prepare_base stage and cleanup_build_stream - Added .gitlab-ci-build-stream.yml with build_stream pipeline support - Added cleanup_build_stream stage to both build-stream and cluster pipelines - Added prepare_base stage to build-stream pipeline (combines repo_manager, image_build_manager, orchestrator setup) - Removed individual deploy stages (repo_manager, image_build_manager, orchestrator) from build-stream - Updated test stages to depend on prepare_base instead of individual deploy stages - Updated summary stage to reflect new pipeline structure - Updated documentation for build_stream pipeline changes - Updated setup_gitlab_project.py for build_stream support Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * fixed variable name Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * updated build_stream pipeline Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add comprehensive build_stream pipeline support to setup scripts and documentation This commit adds full build_stream pipeline support across the Omnia CI/CD pipeline: **setup_gitlab_project.py changes:** - Fixed load_pipeline_config() to read build_stream_enable from pipeline_config.yml - Fixed generate_cluster_build_stream_trigger_job() to include BUILD_STREAM_TAGS variable - Fixed cmd_create() config-file mode to set BUILD_STREAM_ENABLE global default - Ensured all build_stream input files and test configs are collected and committed **pipeline_config.yml changes:** - Added global.build_stream_enable setting to enable build_stream pipeline mode - Added deploy_tags.build_stream for ansible tags configuration - Added test_commands.build_stream with default test command - Updated domains documentation to mention build_stream as valid domain - Updated skip_stages documentation to include build_stream **Documentation additions:** - Created comprehensive BUILD_STREAM_PIPELINE.md guide covering: * Overview and key differences from cluster pipeline * Pipeline stages and conditional execution * prepare_base stage explanation * Configuration reference for all variables * Real-world deployment examples * Input files and credentials setup * Troubleshooting guide - Updated docs/README.md to reference BUILD_STREAM_PIPELINE.md - Added build_stream links to common tasks and getting started sections **Pipeline YAML changes:** - .gitlab-ci-build-stream.yml: Added TEST_BUILD_STREAM_CMD fallback logic - .gitlab-ci-cluster.yml: Added TEST_CMD fallback logic for all test stages - .gitlab-ci-utils.yml: Added TEST_UTILS_CMD fallback logic - .gitlab-ci.yml: Already includes build_stream trigger job generation **Test command fallback mechanism:** All test stages now use bash parameter expansion to fall back to defaults if CI/CD variables are empty or unset, preventing empty command execution. **Report naming:** Summary stages now rename downloaded test reports to include pipeline ID for unique identification across multiple pipeline runs. **Test Execution Summary:** Email notifications now show all domains with appropriate status: - Counts for domains with report data - "No Report" for executed tests without report files - "Not Executed" for test stages that didn't run Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix DRY_RUN report fetching, cleanup_utils status, and old report attachment issues **Issue 1: Test reports being fetched and attached during DRY_RUN** - Added DRY_RUN check before fetching test reports in summary stage - When DRY_RUN=true, report fetching is completely skipped - Prevents unnecessary SSH operations and report attachments during dry runs - Applied to .gitlab-ci-cluster.yml and .gitlab-ci-build-stream.yml **Issue 2: cleanup_utils showing UNKNOWN status despite running** - Added cleanup_utils to summary stage's needs section - Ensures summary stage waits for cleanup_utils to complete before collecting job statuses - Prevents race condition where summary runs before cleanup_utils finishes - cleanup_utils status will now correctly show PASSED/FAILED instead of UNKNOWN - Applied to .gitlab-ci-cluster.yml and .gitlab-ci-build-stream.yml **Issue 3: Old test reports being attached with new pipeline ID prefix** - Removed unsafe Pattern 2 fallback that matched reports without pipeline ID - Now only fetches reports with current pipeline ID to prevent picking up old reports - Pattern 1: Exact match with current pipeline ID - Pattern 2: Broad match with current pipeline ID (was Pattern 3) - This prevents old reports from previous successful runs from being attached when current pipeline fails during initialization - Applied to .gitlab-ci-cluster.yml, .gitlab-ci-build-stream.yml, and .gitlab-ci-utils.yml These fixes ensure: 1. Dry runs don't attach test reports or generate unnecessary artifacts 2. All job statuses are properly captured for email notifications 3. Pipeline execution summary accurately reflects all stage statuses 4. Only reports from the current pipeline are attached to email notifications 5. Old reports from previous runs are never mistakenly attached Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Refactor FVT framework to align with image_build_manager structure - Reorganized FVT tests by playbook tags with test_playbook.py drivers - Split test_status.py files into suite subdirectories (config, pulp, status, cleanup) - Moved catalog tests into separate tag directories (catalog_generate, catalog_add, catalog_delete, catalog_validate) - Moved policy tests into execute/policy/ suite - Distributed negative tests into tag-specific negative/ suites - Removed old catalog/, policy/, and negative/ directories - Updated domain_vars.py with new FVT_TAGS, SUITES, and VERIFY_ONLY_SUITES - Added tl.skipped_fields() to all 51 skip scenarios for professional output - Added pytest_report_teststatus hook to suppress pytest progress characters - Created negative suite under user_registry and moved test_user_registry_negative.py - Updated all __init__.py docstrings and README.md Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * remove the unncessary md files Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix pylint issues in user_registry test_playbook.py - Remove duplicate import of run_playbook - Remove unused import of _get_input_path Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add catalog_validate directory and fix hardcoded catalog path - Create catalog_validate/ directory with validate/ suite - Add test_playbook.py for catalog_validate tag - Add test_validate.py with 3 verification tests: - RM_FVT_CATALOG_VALIDATE_V001: Verify catalog validation completed - RM_FVT_CATALOG_VALIDATE_V002: Verify catalog validation log file exists - RM_FVT_CATALOG_VALIDATE_V003: Verify catalog file still valid after validation - Fix hardcoded catalog path in catalog_generate/generate/test_generate.py - Replace os.environ.get('CATALOG_FILE_PATH', '/opt/omnia/catalog/catalog_rhel.json') - Use _get_catalog_path() function for consistency - Remove unused os import This completes the catalog tag structure and ensures all catalog paths use dynamic configuration functions. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Delete test/repo_manager/update_remaining_skips.py Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Update test_config.yml Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Add discovery stage support and fix pipeline email reporting Major changes: - Add discovery and test_discovery stages to cluster pipeline with conditional execution - Add cleanup_discovery stage to all pipelines (cluster, build-stream, utils) - Remove automation_cluster trigger stages from parent pipeline - Fix test report fetching to include failed tests (not just successful ones) - Fix infrastructure cleanup stages filtering in email reports - Add setup_main and test_main_installation to summary stage needs list - Fix bash syntax errors in summary stage STAGES_EXECUTED variable Pipeline changes (.gitlab-ci-cluster.yml): - Add discovery and test_discovery stages with ENABLE_DISCOVERY conditional - Add cleanup_discovery stage that always runs in cleanup/default modes - Reorder cleanup_utils stage in stage order - Add discovery and test_discovery to summary stage needs - Add setup_main to summary stage needs - Fix bash syntax by using single quotes for multi-line STAGES_EXECUTED - Update SKIP_STAGES documentation to include discovery - Add TEST_DISCOVERY_CMD variable Pipeline changes (.gitlab-ci-build-stream.yml): - Add cleanup_discovery stage (always runs in cleanup/default modes) - Fix test report fetching to include failed tests - Update documentation for discovery support Pipeline changes (.gitlab-ci.yml): - Remove trigger_cluster_automation_cluster (cluster pipeline) - Remove trigger_cluster_automation_cluster_utils (utils pipeline) - Remove trigger_cluster_automation_cluster_build_stream (build-stream pipeline) - Update documentation to remove automation_cluster references - Add ENABLE_DISCOVERY and DISCOVERY_TAGS to cluster1 triggers Email reporting (send_email.py): - Add enable_discovery parameter handling - Add discovery to domain_names and domain_order lists - Add discovery stages to STAGE_ORDER_* lists - Add infrastructure_cleanup_stages filtering (cleanup_build_stream, cleanup_utils) - Add discovery_stages filtering (discovery, test_discovery) - Fix test report fetching to include failed/skipped tests, not just successful ones Documentation updates: - Update README.md with discovery stage information - Update docs/CONFIGURATION.md with discovery configuration - Update docs/PIPELINE_MODES.md with discovery in pipeline flow - Update docs/QUICKSTART.md with discovery examples - Update docs/BUILD_STREAM_PIPELINE.md with discovery notes - Update docs/README.md with discovery references Configuration updates: - Update pipeline_config.yml with discovery-related variables - Update setup_gitlab_project.py for discovery support Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix ansible-lint violations: line length, trailing spaces, and blank lines - Fix line 1864 in .gitlab-ci-cluster.yml: Split long conditional line - Fix line 43 in .gitlab-ci.yml: Remove trailing spaces - Fix line 284 in .gitlab-ci.yml: Remove extra blank line Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> * Fix trailing blank lines in .gitlab-ci.yml Remove extra blank lines at end of file to comply with ansible-lint rules. Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> --------- Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> * fix(test): add missing closing parenthesis in library/functions/__init__.py The import from .hpc_benchmarks_pxeboot_func was missing a closing parenthesis, causing a syntax error that prevented all test files from importing library.functions. Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> * fix(image_build_manager): restore save_compute_image_hashes.yml file Restored the save_compute_image_hashes.yml file that was missing from the pub/rhel10.2 branch. This file is referenced by build_compute_image_aarch64.yml and build_compute_image_x86_64.yml and is required for the build process to work correctly. Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> --------- Signed-off-by: Rajeshkumar S <rajeshkumar.s2@dell.com> Signed-off-by: Rohith Ravut <rohith.ravut@dellteam.com> Signed-off-by: Narthan_S <narthan.s@dell.com> Signed-off-by: snarthan <narthan.s@dell.com> Signed-off-by: balajikumaran-c-s <balajikumaran.cs@dell.com> Signed-off-by: balajikumaran.cs <balajikumaran.cs@dell.com> Signed-off-by: esaifudheen <esaifudheen@dell.com> Signed-off-by: SujalS27 <sujal.sooryavamshi@dell.com> Signed-off-by: SOWJANYAJAGADISH123 <sowjanya.jagadish@dell.com> Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com> Signed-off-by: venu <236371043+Venu-p1@users.noreply.github.com> Signed-off-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> Signed-off-by: VirajPatil-1500 <VirajPatil-1500@users.noreply.github.com> Signed-off-by: Katakam-Rakesh <katakam.rakesh@dell.com> Signed-off-by: Nagachandan P <Nagachandan.p@dell.com> Signed-off-by: Abhishek S A <abhishek.sa3@dell.com> Signed-off-by: VirajPatil-1500 <Viraj.Patil@dellteam.com> Signed-off-by: sayuri <sayuri.kamble@dell.com> Co-authored-by: snarthan <narthan.s@dell.com> Co-authored-by: Rajeshkumar S <rajeshkumar.s2@dell.com> Co-authored-by: Abhishek S A <abhishek.sa3@dell.com> Co-authored-by: Rohith Ravut <rohith.ravut@dellteam.com> Co-authored-by: Sayuri Kamble <68416000+SAYUK09@users.noreply.github.com> Co-authored-by: balajikumaran.cs <balajikumaran.cs@dell.com> Co-authored-by: Sujit Jadhav <sujit.jadhav@dell.com> Co-authored-by: saifudheene <Saifudheen.E@dell.com> Co-authored-by: esaifudheen <esaifudheen@dell.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: SujalS27 <Sujal.Sooryavamshi@dell.com> Co-authored-by: SOWJANYAJAGADISH123 <Sowjanya.Jagadish@dell.com> Co-authored-by: Nagachandan P <Nagachandan.p@dell.com> Co-authored-by: Venu-p1 <236371043+Venu-p1@users.noreply.github.com> Co-authored-by: VirajPatil-1500 <Viraj.Patil@dellteam.com> Co-authored-by: VirajPatil-1500 <VirajPatil-1500@users.noreply.github.com> Co-authored-by: Katakam-Rakesh <katakam.rakesh@dell.com> Co-authored-by: Katakam Rakesh Naga Sai <125246792+Katakam-Rakesh@users.noreply.github.com> Co-authored-by: Vrinda Marwah <vrinda.marwah@dell.com>

* Implement LDMS telemetry enable/disable functionality This commit adds the ability to independently disable and re-enable LDMS telemetry while preserving configuration and data, without affecting other enabled telemetry sources. Changes: - Remove conditional execution of LDMS role from main deployment flow - Add playbook-level check to disable LDMS when metrics_enabled is false - Enhance disable.yml to handle all LDMS components (aggregator, store, sampler services, Vector-LDMS bridge) - Enhance enable.yml for re-enablement with state preservation and PVC UID verification - Add password generation and validation for LDMS authentication to fix empty password issue - Update status reporting to show 'disabled' instead of 'skipped' - Add idempotency checks for repeated disable operations The implementation ensures that when telemetry_sources.ldms.metrics_enabled is false, all LDMS components are scaled to zero while preserving PVCs, secrets, Helm metadata, and configuration. When re-enabled, components are restored and existing PVCs are reused with unchanged UIDs. Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Implement LDMS enable/disable functionality with NFT test coverage - Centralize LDMS sampler password generation in deploy_ldms.yml to prevent authentication failures between aggregator and sampler - Add | bool to all when conditions to fix boolean string evaluation issues - Replace hardcoded Helm release name with ldms_aggregator_name variable - Add conditional password retrieval when LDMS is disabled to prevent errors - Remove random password generation - LDMS sampler password must be user-provided in telemetry_credentials (security requirement) - Implement comprehensive NFT test for LDMS configuration transitions - Tests 4 states: baseline, Vector disabled, both disabled, restored - Validates LDMS aggregator scaling across all transitions - Includes helper function get_pod_count_by_prefix for pod counting - Fix pylint issues: remove trailing whitespace, fix import order, remove unused imports, fix exception handling, reduce statement count - Fix ansible-lint line too long errors by breaking long when conditions - Add test case ID TEL_NFT_LDMS_001 for LDMS configuration transitions Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Fix: Add ldms_sampler_password to all plays that need it - Add ldms_sampler_password to Prepare LDMS Authentication play - Add ldms_sampler_password to Configure LDMS Sampler play - Password is exposed by telemetry_prereq.yml and copied from localhost - Fixes: ldmsauth.conf had empty password causing ldmsd to fail Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Reduce end_play guards from 5 to 1 using block+when pattern Changes: 1. deploy_ldms.yml: Replace meta: end_play in Plays 2-5 with block+when - Play 1 keeps meta: end_play (primary gate for disabled state) - Plays 2-5 use block with when condition to skip tasks when disabled - Reduces from 5 end_play guards to 1 - Uses | default(false) | bool for safer variable handling 2. disable.yml: Add pause after scaling operations - Add 30s pause after LDMS aggregator scale down - Add 30s pause after LDMS store scale down - Add 30s pause after Vector-LDMS scale down - Ensures pods fully terminate before status checks - Fixes timing issue where status reported 'skipped' instead of 'disabled' 3. enable.yml: Prevent unnecessary pod restarts - Get current replica count before scaling - Only scale if current != target - Prevents restarts when already at desired state - Add Vector-LDMS scale down when disabled (even if LDMS enabled) 4. deploy.yml: Enhanced status reporting logic - Add kubectl checks for StatefulSet/Deployment existence - Distinguish between deployed/disabled/skipped/failed - deployed: enabled and running - disabled: disabled but resources exist (scaled to 0) - skipped: disabled and resources don't exist (never deployed) - failed: state doesn't match configuration 5. test_ldms_config_transitions.py: Add status verification - Add _get_status_path() helper (fixed path resolution) - Add _verify_status_reporting() function - Parse telemetry_status.yml for LDMS and Vector-LDMS status - Verify status matches expected state for each transition - Update docstring to reflect status verification Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Fix ansible-lint issues: line length, idempotency, and undefined variable Changes: 1. deploy.yml: Break long Jinja2 lines into multi-line blocks - LDMS status logic: split into if/else blocks with trim markers - Vector-LDMS status logic: split into if/elif/else blocks with trim markers - Added | trim filter to remove whitespace from status values - Lines now under 160 characters as required by ansible-lint 2. disable.yml: Fix idempotency and undefined variable issues - Set changed_when: false for all scale commands (kubectl scale always reports 'scaled') - Fixed undefined 'sampler_stop_result' variable in ldms_disable_changes_made - Removed sampler_stop_result from change detection (sampler services not stopped) - Updated log message to remove sampler reference Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Fix ansible-lint: add changed_when/failed_when to Vector-LDMS check The 'Check Vector-LDMS deployment existence' command task was missing changed_when: false and failed_when: false, causing a no-changed-when ansible-lint violation. All other command tasks in the status check section already had these attributes. Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Refactor LDMS telemetry deployment to 2-play structure and fix restart bugs Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Address review comments and improve idempotency - Add replica count checks before kubectl scale to eliminate cosmetic "changed" false-positives on idempotent re-runs. - Remove VictoriaMetrics deployment from the LDMS playbook; it is deployed centrally in the sinks phase, not per telemetry source. - Remove redundant ldms_sampler_password propagation from the playbook; it is already set/read where actually consumed. - Replace manual telemetry_config/mounts propagation in Play 2 with common/load_telemetry_config include. - Remove disable+end_play short-circuit from the playbook; main.yml already routes to disable.yml internally. - Move build_deployment_inventory include into enable.yml, since it is only required for enable, not disable. - Extract Vector-LDMS bridge scale-down into a dedicated reconcile_vector_ldms_bridge.yml with an explanatory comment. - Add clarifying header comment to deploy_vector_ldms.yml. Verified with yamllint and ansible-lint (production profile), both clean, and a full NFT test (TEL_NFT_LDMS_001, 962s) covering all 4 enable/disable/re-enable transitions. Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> * Address review comments: clarify deploy_vector_ldms.yml call chain - Updated header comment in deploy_vector_ldms.yml to explicitly show the complete call chain from playbook to task file - Added clarification that it handles both deployment and disable (scale to desired replicas vs scale to 0) Call chain documented: deploy_ldms.yml (Play 1) → enable.yml → reconcile_vector_ldms_bridge.yml → deploy_vector_ldms.yml Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> --------- Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> Signed-off-by: Sushmitha U <Sushmitha.U@dell.com>

* Add enable/disable/re-enable flow for UFM telemetry Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Fix UFM disable: remove when gate and add VMServiceScrape cleanup Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Add FVT test for UFM disabled state verification Add test_ufm_disabled_state (TEL_FVT_DEPLOY_V064) to verify that when UFM telemetry is disabled, the Service, Endpoints, and VMServiceScrape are removed while the credentials Secret is preserved for re-enablement. The test follows the same pattern as idrac/ome/powerscale disabled state tests but checks for resource absence instead of workload scale-down (since UFM is external and has no pods to scale). Changes: - Add verify_ufm_resources_absent() helper in ufm_func.py - Add test_ufm_disabled_state in test_ufm.py - Add disabled state messages in ufm_msgs.py - Add ufm_disabled_state test case entry in test_case_vars.py Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> * Update deploy.yml Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> * pylint fixes Signed-off-by: Kratika-P <Kratika.Patidar@dell.com> * removing load input file redundant task from enable and disable ufm Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> --------- Signed-off-by: Kratika_Patidar <Kratika.Patidar@dell.com> Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com> Signed-off-by: Kratika-P <Kratika.Patidar@dell.com>

…erns for deploy_sinks and per-source cleanup tags (#5476) * Standardize Telemetry Test Framework with Cross-Domain Patterns Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update Readme files Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Remove empty unused files Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> --------- Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>

…dd selective sink configuration, and fix failing tests (#5488) * Standardize Telemetry Test Framework with Cross-Domain Patterns Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update Readme files Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Remove empty unused files Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Add Test Config Variables for Selective Sink Deployment and Cleanup Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> * Update test_config.yml Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com> --------- Signed-off-by: Mithilesh Reddy <mithilesh.reddy@dell.com>

* Issue 776: Fix TEL_FVT_DEPLOY_V038 false-negative on healthy zero-event PowerScale deployment Updates TEL_FVT_DEPLOY_V038 to distinguish continuously available state/resource metrics from event-conditioned metrics. Event-conditioned series are only required when matching Kubernetes events exist. The test is skipped when the CSI volume exporter is intentionally not deployed because external-health-monitor-controller is unavailable. Signed-off-by: U, Sushmitha <Sushmitha_U@Dell.com> * Fix LDMS sampler configuration file transfer and optimize node reachability This commit addresses the root cause of 'Source not found' errors by delegating sampler config preparation to the Slurm control node. Problem: - Files were created on kube_vip's NFS mount - Compute nodes couldn't see them immediately due to NFS sync delays - configure_ldms_sampler.yml tried to copy files that didn't exist Solution: 1. Delegate prepare_ldms_sampler_config.yml tasks to Slurm control node - Files are now created directly on control node's NFS mount - All compute nodes access the same NFS mount point - Files are immediately visible to all nodes 2. Optimize node reachability check - Use sinfo -r flag to filter responsive nodes natively - O(1) performance instead of O(n) SSH checks - 200x faster for large clusters 3. Fix unreachable hosts calculation - Use difference() to calculate unreachable nodes - Proper error reporting and failure handling Changes: - Added delegate_to: slurm_control_node to prepare_ldms_sampler_config.yml - Removed unnecessary NFS verification and sync tasks - Simplified configure_ldms_sampler.yml (uses NFS directly) - Changed shell to command module for sinfo - Fixed unreachable hosts using difference() - Centralized sinfo_command in vars/main.yml Impact: - Fixes 'Source not found' error - Eliminates NFS synchronization issues - Improves performance (O(1) node reachability) - Cleaner, simpler code Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> --------- Signed-off-by: U, Sushmitha <Sushmitha_U@Dell.com> Signed-off-by: Sushmitha-dell <Sushmitha.u@dell.com> Co-authored-by: U, Sushmitha <Sushmitha_U@Dell.com>

Kratika-P and others added 7 commits September 23, 2026 17:56

Kratika-P requested review from RvishankarOMnia, VenkateswaraVatam, abhishek-sa1, priti-parate and snarthan as code owners September 30, 2026 12:02

Kratika-P changed the title ~~sync Pub/issue 4849 telemetry to staging~~ Sync Pub/issue 4849 telemetry to staging Sep 30, 2026

Kratika-P changed the title ~~Sync Pub/issue 4849 telemetry to staging~~ Sync pub/issue 4849 telemetry to staging Sep 30, 2026

abhishek-sa1 approved these changes Sep 30, 2026

View reviewed changes

abhishek-sa1 and others added 9 commits September 30, 2026 19:01

Merge branch 'staging' into pub/issue_4849_telemetry

e8e7909

correct when condition indentation in PowerScale debug task (#5422)

b8af15c

Signed-off-by: Kratika Patidar <Kratika.Patidar@dell.com>

Merge branch 'staging' into pub/issue_4849_telemetry

bd70004

priti-parate requested review from Rajeshkumar-s2, Venu-p1, saifudheene and sujit-jadhav as code owners October 6, 2026 19:20

Sushmitha-dell and others added 2 commits October 7, 2026 10:55

mithileshreddy04 and others added 3 commits October 8, 2026 15:49

Navigation Menu

Repository navigation

Sync pub/issue 4849 telemetry to staging - #5415

Sync pub/issue 4849 telemetry to staging#5415
Kratika-P wants to merge 21 commits into
stagingfrom
pub/issue_4849_telemetry

Kratika-P commented Sep 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Kratika-P commented Sep 30, 2026

PR Title

Description of the Solution

Related Issue

Changes

Component (path/to/files)

Files Changed

Testing

Backward Compatibility

Suggested Reviewers

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Component (`path/to/files`)