Memory stream pulling agent cannot dequeue queue grain — message rejected after 'Unable to create local activation'

[systemorph-com]

What is failing

The Orleans persistent-stream pulling agent on the memex portal pod cannot dequeue from the in-memory stream queue memory-2-0x40000002: its Dequeue message to the MemoryStreamQueueGrain on silo 10.244.9.223 was rejected with OrleansMessageRejectionException ("Forwarding failed … after 'Unable to create local activation'"), and MemoryAdapterReceiver.GetQueueMessagesAsync failed on its first retry.

Probable cause

Cluster-membership churn at the time (silos restarting or unstable) left the queue grain's activation unavailable and Orleans could not re-create it locally, so the message was rejected after 2 forwarding attempts. Confidence: moderate — the exception text itself states the activation failure, but this incident carries a single sample and I cannot confirm from the log alone which silo event (restart, eviction, activation-limit rejection) preceded it. The same window shows sibling incidents on the same and neighbouring queues (SiloUnavailableException, TimeoutException in MemoryAdapterFactory.GetQueueMessagesAsync) plus a fleet-wide DbVersionGate/hosting-start failure, all pointing at an unstable cluster rather than a defect in the streaming code path.

Impact

Minimal on its own: 1 occurrence on 1 pod, no recurrence in the ~8 hours since. Note that a rejected dequeue on a memory stream can drop that queue's messages — memory streams are not durable — so if these rejections become frequent during normal operations, stream events would be silently lost.

Where to look

Start at Orleans.Streaming.PersistentStreams.PersistentStreamPullingAgent.ReadFromQueue (/_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs, line 572) and Orleans.Providers.MemoryAdapterReceiver.GetQueueMessagesAsync (line 45); correlate with silo membership events for 10.244.9.223 around 2026-09-25 20:59 UTC. Likely a duplicate/symptom of the same cluster-instability cluster as incidents Admin/_LogIncident/c38f54cff9081656 and Admin/_LogIncident/2fe0d61347f3b619 — check those before treating this as a distinct defect.

---

Evidence

Fingerprint	935c6e45a7ce7ed9
Category	Orleans.Streams.Memory
Severity	Error
Exception	Orleans.Runtime.OrleansMessageRejectionException
Namespace	memex
Pods	memex-portal-deployment-56fbdcd48f-cbms9
Occurrences	1
First seen	2026-09-25 20:59:03Z
Last seen	2026-09-25 20:59:03Z
Routing	not determined — no configured route matches the category Orleans.Streams.Memory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-25 20:59:03Z memex-portal-deployment-56fbdcd48f-cbms9 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-2-0x40000002
      Orleans.Runtime.OrleansMessageRejectionException: Forwarding failed: tried to forward message Request [S10.244.3.247:11111:149367510 sys.svc.stream.agent/10.244.3.247:11111@149367510+Memory_5_memory-2-0x40000002]->[S10.244.9.223:11111:149363880 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #75D24AA699D77E3C[ForwardCount=2] for 2 times after "Unable to create local activation" to invalid activation. Rejecting now. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

_{Re-addressed by the current identity function: this incident inherited c38f54cff9081656 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/935c6e45a7ce7ed9. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 67be4a4c48b69519: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}