fix(orchestrator): renew OpenCHAMI tokens before use

[balajikumaran-c-s]

PR Title

fix(orchestrator): renew OpenCHAMI tokens before use

Description of the Solution

metadata-service and boot-service exchange a single-use TokenSmith bootstrap token at startup for a refresh-token family that expires after 24 hours. The services refresh access tokens at runtime, but refresh-token rotation does not extend the family's fixed expiry. This change restarts the two services whenever deploy_openchami runs and once in the provision preamble, so deploy-only and provision-only runs receive new bootstrap credentials. HAProxy restarts afterward to resolve their new container addresses.

This is a bounded Omnia recovery workaround. It does not use service uptime as a proxy for token validity and does not introduce a permanent timer. The long-term credential lifecycle remains an upstream OpenCHAMI packaging/service concern.

Related Issue

• Related to [Feature request] Omnia Modernization #4849

Root Cause

• TokenSmith v0.4.1 rotates refresh tokens while retaining the refresh family's original ExpiresAt.
• The OpenCHAMI RPM helper creates bootstrap policies with TokenSmith's default 24-hour refresh lifetime.
• After family expiry, the one-time bootstrap token cannot be reused; restarting the service runs its packaged ExecStartPre helper and obtains fresh credentials.

Changes

• Added renew_service_tokens.yml to restart metadata-service.service and boot-service.service, followed by haproxy.service.
• Included renewal after the existing certificate-renewal configuration on every deploy_openchami run.
• Included the same renewal once in provision_preamble.yml, before OpenCHAMI authentication, so --tags provision and --tags execute are covered.
• Documented the deployment-time and pre-provision restart behavior.
• No custom systemd service/timer, uptime logic, cleanup changes, new variables, or OpenCHAMI source changes are included.

Files Changed

File	Change Type	Description
src/orchestrator/roles/deploy_openchami/tasks/renew_service_tokens.yml	Added	Restarts the token-consuming services and HAProxy
src/orchestrator/roles/deploy_openchami/tasks/main.yml	Modified	Runs token renewal during deployment
src/orchestrator/playbooks/provision/provision_preamble.yml	Modified	Renews credentials once before OpenCHAMI authentication
src/orchestrator/roles/deploy_openchami/README.md	Modified	Documents deployment-time and pre-provision renewal

Testing

• ansible-lint: passed with 0 failures and 0 warnings at the production profile.
• ansible-playbook --list-tasks --tags provision confirms the renewal play and include carry both provision and execute tags.
• Ran the new task file against the lab OIM: metadata-service, boot-service, and HAProxy restarted successfully with no task failures.
• Verified all three systemd services are active after the restart.
• Verified /metadata-service/health and /boot-service/health return HTTP 200 through HAProxy.
• The full deploy_openchami.yml playbook was not run.

Backward Compatibility

• No new inputs or persistent systemd units.
• Each deployment or provision run briefly interrupts the OpenCHAMI APIs while the three services restart.
• Tokens can still expire between operator runs; deployment and provisioning now recover credentials before using the services, but this is not continuous renewal.

Suggested Reviewers

Reviewer	Review Focus
@abhishek-sa1 @VenkateswaraVatam	deploy_openchami behavior and bounded workaround
@sujit-jadhav	OpenCHAMI and TokenSmith credential lifecycle

[VenkateswaraVatam]

The health validation and recovery handling look useful, but I have concerns about using service uptime and service restart as the mechanism for TokenSmith credential renewal. TokenSmith's service-authentication model supports refresh-token based renewal and automatic token refresh. Can we first confirm whether the current Fabrica metadata-service/boot-service implementations support or are expected to support this lifecycle? If refresh is missing in those services, this looks like an upstream OpenCHAMI/Fabrica gap, and the restart mechanism should preferably be treated as a bounded Omnia workaround rather than the long-term token renewal design. We should also avoid coupling service uptime directly to token validity where possible.

[VenkateswaraVatam]

Since we are using the Fabrica-based metadata-service and boot-service, can we confirm why these services require a restart to renew their TokenSmith credentials? TokenSmith's bootstrap-first service flow returns both an access token and refresh token and is designed to maintain refresh state.
The preferred solution should be for the Fabrica services to manage their credential lifecycle rather than Omnia restarting healthy services based on systemd uptime.
I am fine with retaining the health check and a restart as a recovery mechanism for an unhealthy service, but I would avoid making the 18-hour service restart the permanent token-renewal mechanism. If the Fabrica services currently do not refresh credentials at runtime, we should track that as an upstream OpenCHAMI/Fabrica gap and treat the restart here as a temporary workaround.

[balajikumaran-c-s]

@VenkateswaraVatam Thank you for the feedback. I investigated the TokenSmith configuration and found that it DOES support configurable refresh token lifetimes via the --refresh-ttl flag (min 1h, max 30d, default 24h). The issue is that the OpenCHAMI RPM's tokensmith_bootstrap_token script does NOT pass --refresh-ttl , so it uses the default 24h. The services do have auto-refresh code and are actively using it, but the refresh token family itself expires after 24 hours. The proper fix is to update the OpenCHAMI RPM to pass --refresh-ttl 7d or --refresh-ttl 30d when creating bootstrap tokens. This is an upstream OpenCHAMI/Fabrica change. The timer-based restart in this PR is a temporary workaround until the upstream RPM is updated. Once the upstream fix is available, we can remove the timer from Omnia.

[VenkateswaraVatam]

Based on this clarification, I support merging PR #5498 as a temporary mitigation. Please document the 24-hour TokenSmith refresh-family lifetime as the root cause and link an upstream OpenCHAMI RPM issue to explicitly configure --refresh-ttl.

The current behavior should be described as service re-bootstrap through restart, not token refresh. Please also define when the 18-hour timer can be removed and validate the workaround against actual refresh-family expiry. Since even a 30d TTL eventually expires, we should separately track automatic refresh-family renewal or re-bootstrap as the long-term solution.

Based on the clarification, I am fine with merging PR #5498 as a temporary mitigation, with the following follow-up actions:

1. Document the root cause as the 24-hour TokenSmith refresh-token family lifetime, resulting from the OpenCHAMI RPM relying on the default --refresh-ttl value.
2. Create and link an upstream OpenCHAMI RPM issue to explicitly configure --refresh-ttl, preferably through a configurable setting rather than a hard-coded value.
3. Describe the current recovery accurately as service re-bootstrap through restart, rather than token refresh.
4. Define a clear removal condition for Omnia’s 18-hour timer once the upstream lifecycle fix is available and validated.
5. Test the workaround against actual refresh-family expiry, not only service uptime.
6. Track a long-term requirement for automatic refresh-family renewal or re-bootstrap because extending the TTL to 30d postpones expiry but does not permanently resolve the credential lifecycle.