fix: recover SOAP requests after Dataverse BCDR redirects

[Suyash Kumar Patel (suyash1208)]

Summary

Dataverse BCDR can redirect the SOAP organization endpoint from the canonical organization host to the same organization's temporary --d host. WCF follows that cross-host redirect but clears the Authorization header, so the temporary host returns 401 and ServiceClient currently treats the request as a terminal failure.

This change uses that final authentication failure as a bounded recovery signal for ExternalTokenManagement connections.

Changes

• Detect the final redirected authority from the HTTP response or WWW-Authenticate challenge.
• Accept only HTTPS transitions between the same organization's canonical and exact --d host, with the same DNS suffix and port and no user information.
• Request a token for the redirected service URI through the existing external token callback.
• Recreate the WCF organization proxy while preserving caller settings, timeout behavior, custom proxy-type assemblies, and request handlers.
• Retry the original request once after the trusted endpoint and token are installed.
• Share the learned endpoint across a root ServiceClient and its clones, including the reverse transition back to canonical when Dataverse redirects it.
• Add focused validation tests and a release-note entry.

Scope and safety

• Recovery is limited to AuthenticationType.ExternalTokenManagement.
• Arbitrary hosts, different organizations, different DNS suffixes, non-HTTPS targets, port changes, user-info URIs, and routing suffixes other than --d are rejected before requesting a token.
• The retry occurs only after the automatically redirected request reaches the trusted target without authorization and receives an authentication failure.

Validation

• 14 focused cross-host redirect tests passed on net8.0.
• Full core unit suite: 52 passed, 27 live-connection tests skipped.
• net462 ServiceClient build succeeded with 0 warnings and 0 errors.

[abelmilash-msft]

Possible deadlock: this sync-path recovery call isn't wrapped in Task.Run like the retry just below.

Unlike the async overload (which awaits this call at line 1913), here .GetAwaiter().GetResult() blocks the calling thread while TryRecoverFromCrossHostRedirectAsync invokes the consumer-supplied GetAccessTokenAsync. On a thread with a SynchronizationContext (WinForms/WPF/classic ASP.NET), if that callback resumes on the captured context it deadlocks against the blocked caller.

The retry a few lines below already avoids exactly this by offloading to the thread pool via Task.Run(...). Suggest matching that pattern here:

Suggested change

	bool redirectRecovered = !crossHostRedirectRecoveryAttempted &&
	_connectionSvc.TryRecoverFromCrossHostRedirectAsync(ex, requestServiceUri, requestTrackingId).ConfigureAwait(false).GetAwaiter().GetResult();
	bool redirectRecovered = !crossHostRedirectRecoveryAttempted &&
	Task.Run(() => _connectionSvc.TryRecoverFromCrossHostRedirectAsync(ex, requestServiceUri, requestTrackingId))
	.ConfigureAwait(false).GetAwaiter().GetResult();

[abelmilash-msft]

Untested branch: the WebException/HttpWebResponse 401 path (including this ResponseUri authority) has no coverage.

The new recovery tests only construct MessageSecurityException (ServiceClientTests.cs:284, 326), so this branch is never exercised. Since TryGetRedirectAuthority is private static, a test would drive a WebException wrapping a fake HttpWebResponse through TryRecoverFromCrossHostRedirectAsync.

Worth covering because recovery here hinges on ResponseUri being the final, post-redirect URI (the --d host). If it is ever the original request URI, TryCreateTrustedRedirectServiceUri(current, current) sees the same host and rejects it, so recovery silently no-ops.

Suggest adding a test for (a) ResponseUri = https://contoso--d.crm.dynamics.com/... returning the --d authority, and (b) a null-ResponseUri case that falls back to the WWW-Authenticate resource_id.

[abelmilash-msft]

Bug: this can crash a concurrent request with ObjectDisposedException.

Recovery swaps in a new proxy and then disposes the old one here. But another thread on the same ServiceClient may still be using that old proxy inside Execute. The request path locks on _lockObject; this recovery uses a different lock (_redirectRecoveryLock) and disposes outside it, so the two do not coordinate and the in-flight request gets ObjectDisposedException.

I reproduced this with a small standalone program using the same swap-and-dispose pattern: the current approach throws ObjectDisposedException under concurrency, and not disposing eliminates it.

Simplest fix — do not dispose the old proxy; drop the reference and let GC reclaim it. The active proxy is still disposed normally at end-of-life (Dispose()), so this only defers cleanup of the one proxy a rare failover swaps out.

Suggested change

	DisposeWebProxy(previousProxy);

Keep the catch-path DisposeWebProxy(replacementProxy) — it disposes only the new, not-yet-published proxy on failure, which no other thread can hold. If deterministic disposal is required, instead do the swap+dispose under _lockObject so no request is in flight (more involved, since recovery is async).

[abelmilash-msft]

Untested branches: recovery-already-completed race, empty-token, and token-acquisition-throws paths.

Only the "happy path" (fresh trusted redirect + successful token) and the different-org rejection are covered by CrossHostRedirect_RecoveryReplacesProxyAndRequestsRedirectToken / CrossHostRedirect_RecoveryRejectsDifferentOrganization (ServiceClientTests.cs:~284, ~326). Three other branches inside this method have no test coverage:

1. recoveryAlreadyCompleted (lines 3785-3793) - a second concurrent request hitting the redirect after another request already swapped the endpoint. Returns true without re-requesting a token; worth a test asserting GetAccessTokenAsync is not invoked a second time.
2. Empty/null token from GetAccessTokenAsync (lines 3821-3830) - falls back to false (no recovery), leaving the original request to fail as before. Worth asserting recovery is a no-op rather than throwing.
3. GetAccessTokenAsync throwing (lines 3832-3840) - caught and logged, recovery returns false. Worth asserting the exception doesn't propagate and the caller's original exception still surfaces.

Since TryRecoverFromCrossHostRedirectAsync is internal, these are all reachable via InternalsVisibleTo from the existing test project using a fake GetAccessTokenAsync delegate.

[abelmilash-msft]

Untested: "recovery succeeded but the retried request still fails" scenario (both async and sync loops).

Both redirectRecovered branches here (and the mirrored one in the sync Execute overload around line 2047) set retry = true and loop back to re-issue the request against the new endpoint - but neither existing CrossHostRedirect_* test drives an actual retry through the outer while loop; they call TryRecoverFromCrossHostRedirectAsync directly and assert on its return value/side effects, not on the surrounding retry orchestration.

Worth a test where the retried request (post-recovery) still throws (e.g. the --d failover host is itself unreachable, or its own token validation still rejects). Since crossHostRedirectRecoveryAttempted is set to true before the retry, the interesting assertion is that the second failure does not attempt cross-host recovery again (falls through to ShouldRetry/normal retry-or-throw handling) rather than looping indefinitely or masking the second exception.