Memory stream dequeue on queue memory-5 times out when the remote silo's MemoryStreamQueueGrain stalls (>30s no response)

[systemorph-com]

What is failing
The Orleans memory-streaming receiver in the memex portal fails to pull messages: its grain call IMemoryStreamQueueGrain.Dequeue to a remote silo's queue grain does not return within the default 30-second response timeout, and MemoryAdapterReceiver.GetQueueMessagesAsync (src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:45, Orleans 10.0.0) logs the TimeoutException. Delivery from the memory stream queue stalls for as long as the target silo stays unresponsive.

Probable cause
Most likely a transient stall or activation delay of the dequeue grain on the target silo, not a code defect in the portal: the request crossed silo boundaries (10.244.2.41 -> 10.244.3.152, later 10.244.3.247 -> 10.244.2.175) and the queue grain simply did not answer in time, then recovered. Confidence: medium-low — only 4 occurrences over ~6.5 hours, never recurred since 2026-09-24, and the top frames are all framework code. Note that sibling incidents for the same component fired across queues memory-0…memory-7 with TimeoutException and SiloUnavailableException in the same window (see Admin/_LogIncident/df0abaed632df2a5, Admin/_LogIncident/0e9b05c4e0b6c4ac, Admin/_LogIncident/bc82a1cb61601923), which suggests one common cluster event on the memex deployment rather than eight separate defects — please dedupe against those before doing per-queue work.

Impact
Minimal: 4 occurrences across 2 pods over ~6.5 hours, two days ago. Streaming delivery on queue memory-5 stalled briefly and self-recovered (the receiver retries on its next poll cycle). No other queues or pods show sustained impact.

Where to look
Start at MemoryAdapterReceiver.GetQueueMessagesAsync — the receiver swallows the timeout and retries on its next poll, so the real question is what made the target silo's IMemoryStreamQueueGrain unresponsive for >30s (silo GC pause, activation backlog, or a wedged read hub, as seen in the 2026-09-21 memex restart action Ops/Actions/restart-memex-wedged-reads-20260921). If nothing similar recurs, this can likely be closed as transient.

---

Evidence

Fingerprint	4009d098b8e7b4f7
Category	Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-5`
Severity	Error
Exception	System.TimeoutException
Namespace	memex
Pods	memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-56fbdcd48f-cbms9
Occurrences	4
First seen	2026-09-24 06:26:15Z
Last seen	2026-09-24 12:52:44Z
Routing	not determined — no configured route matches the category Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-5`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-24 06:26:15Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-5[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_3_memory-5-0xA0000005]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F876B63. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
2026-09-24 06:26:56Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-5[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_3_memory-5-0xA0000005]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F8767A2. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-5[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_3_memory-5-0xA0000005]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A68. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
2026-09-24 12:52:44Z memex-portal-deployment-56fbdcd48f-cbms9 fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-5[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.3.247:11111:149258867 sys.svc.stream.agent/10.244.3.247:11111@149258867+Memory_3_memory-5-0xA0000005]->[S10.244.2.175:11111:149249283 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #0FFBB7600D992476. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45

_{Re-addressed by the current identity function: this incident inherited d34beec418034031 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/4009d098b8e7b4f7. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 99e47a72c57e984d: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}