Silent permanent event-processing stall mid-batch (unobserved Task exception, no error/crash)

[diogodias8x8]

This issue was drafted with the help of an AI coding assistant (Claude), based on live reproduction and source-code analysis I ran and reviewed myself.

Environment

• Reflector: 10.0.61 (docker.io/emberstack/kubernetes-reflector:10.0.61)
• Kubernetes: v1.35.6-eks-8f14419 (AWS EKS)
• Deployment: standard chart, single replica, configuration.logging.minimumLevel: Verbose for this investigation

How this differs from prior closed issues

I searched the tracker first — #77, #88, #239, #341, #442, #467, #511, #545, #657 all describe superficially similar symptoms ("stops reflecting, restart fixes it"), but every one I read shows a logged failure: a Faulted due to exception line with an HTTP2/IOException stack trace, or a repeating Session closed / Requesting resources cycle. Those are watch-connection-level failures caught by WatcherBackgroundService.ExecuteAsync's outer catch (Exception exception) block (Faulted due to exception. + sessionFaulted = true), which then loops and re-establishes a new session — sometimes successfully, sometimes not, per those issues.

What I'm reporting is different and, as far as I can tell, not covered by any of those: zero log output of any kind, at Verbose level, from the moment it stops — not even a Session closed line. Root cause below.

Reproduction

kubectl -n kube-system create secret docker-registry probe \
  --docker-server=x --docker-username=x --docker-password=x
kubectl -n kube-system annotate secret probe \
  reflector.v1.k8s.emberstack.com/reflection-allowed=true \
  reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces=".*" \
  reflector.v1.k8s.emberstack.com/reflection-auto-enabled=true \
  reflector.v1.k8s.emberstack.com/reflection-auto-namespaces=".*"

# wait for the ~70-namespace fan-out to complete (succeeds every time)

kubectl -n kube-system delete secret probe
# → triggers deletion across all ~70 reflected namespaces

On a cluster with ~70 namespaces, this reliably wedges the SecretWatcher's processing loop mid-deletion. Reproduced 3 separate times across two sessions.

Observed (Verbose logging, timestamps included)

Deletion proceeds in strict alphabetical order, one [DBG] line per namespace, ~12-14ms apart:

13:56:28.916 [VRB] (SecretMirror) Handling Deleted Secret kube-system/diogo-reflector-write
13:56:28.916 [DBG] (SecretMirror) Deleting alert-8x8-chat/diogo-reflector-write - Source ... has been deleted
13:56:28.949 [DBG] (SecretMirror) Deleting arc-runners-buildkit/diogo-reflector-write - ...
...
13:56:29.343 [DBG] (SecretMirror) Deleting kargo-shard-test-eght-en1-02/diogo-reflector-write - ...
13:56:29.360 [DBG] (SecretMirror) Deleting kargo-system-resources/diogo-reflector-write - ...

Then: nothing. No further Deleting lines for the remaining namespaces (37 of 71 secrets were left behind), no Session closed, no [ERR], no exception, at any level, ever again. Pod remains 1/1 Running, readinessProbe/livenessProbe green, 0 restarts. Only kubectl rollout restart recovers it.

Root cause

src/ES.Kubernetes.Reflector/Watchers/Core/WatcherBackgroundService.cs#L54-L75:

//Read using a separate task so the watcher doesn't get stuck waiting on subscribers to handle the event
_ = Task.Run(async () =>
{
    try
    {
        while (!cancellationToken.IsCancellationRequested)
        {
            var watcherEvent = await eventChannel.Reader.ReadAsync(cancellationToken).ConfigureAwait(false);
            foreach (var watcherEventHandler in watcherEventHandlers)
                await watcherEventHandler.Handle(new WatcherEvent { ... }, cancellationToken);
        }
    }
    catch (OperationCanceledException)
    {
        // Expected on session shutdown when cancellation propagates through API calls.
    }
}, cancellationToken);

_ = Task.Run(...) discards the returned Task — nothing observes it. The catch only handles OperationCanceledException. Any other exception thrown inside watcherEventHandler.Handle(...) is unobserved; under .NET (Core 3.0+), an unobserved faulted Task does not crash the process by default — it's simply dropped. The while loop, and with it all future event processing for that resource type, ends silently, forever, in that pod's lifetime.

Prime suspect for the actual throw site: src/ES.Kubernetes.Reflector/Mirroring/Core/ResourceMirror.cs#L421-L455, AutoReflectionForSource — unguarded calls at L430, L431-L432, and L455:

var matches = await OnResourceWithNameList(sourceNsName.Name, cancellationToken);      // unguarded
var namespaces = (await Kubernetes.CoreV1.ListNamespaceAsync(cancellationToken: cancellationToken)).Items;  // unguarded
...
foreach (var reference in toDelete) await OnResourceDelete(reference);                  // unguarded

None of these three calls — nor anything in HandleUpsert or Handle above them — has a try/catch. Only the innermost per-target ResourceReflect call is guarded (it has an explicit catch (HttpOperationException ... Conflict) and a broad catch (Exception ex) { LogError(...) }). A transient HttpOperationException (429, timeout, or a resourceVersion-related list error) from either of the two unguarded list calls above — plausible under the API-server load of a ~70-namespace batch — would propagate all the way up, out of Handle(), and hit the fire-and-forget task's narrow catch, silently ending everything.

Suggested fix

Either (a) wrap AutoReflectionForSource's body in a try/catch that logs and returns rather than throws, or (b) change the fire-and-forget task's catch to catch (Exception ex) when (ex is not OperationCanceledException) { logger.LogCritical(ex, "Unobserved exception in watcher event processing loop"); } so a future occurrence is at least visible, or ideally both.

Happy to provide the full log capture or test further against a specific branch/build if useful.

[komapa]

Do you want to try this patch to see if it works for you #646?