What is failing. The Orleans memory-stream receiver for queue memory-1 on the memex portal pod (memex-portal-deployment-745745db7c-b2gjn) timed out waiting for IMemoryStreamQueueGrain.Dequeue on a peer silo (10.244.5.80 → 10.244.2.175): the grain did not answer within the 30 s response bound, so MemoryAdapterFactory.GetQueueMessagesAsync threw and the receiver's polling cycle for that queue faulted. This is the in-process streaming data plane (the per-silo cache/sync-stream transport behind IMeshNodeStreamCache and the change feed), not application code.
Probable cause — moderate confidence. The receiver itself is fine; the failure is on the far side: the MemoryStreamQueueGrain hosting that queue was unresponsive for over 30 seconds. That is consistent with this fleet's known history of wedged silo-side actors under load — see the memex restart actions Ops/Actions/restart-memex-wedged-reads-20260921 ("wedged read hubs") and restart-memex-confirmed-20260921 (stale enumeration snapshot pumping a deleted partition) from three days before this burst. A grain stuck behind a long turn (e.g. a blocked hub round) would answer late or never within the response bound. The evidence does not positively identify which turn wedged the grain — cause at the grain level unclear; evidence points at silo-side congestion/wedged-actor contention on 10.244.2.175.
Impact. Minimal as measured: a single occurrence on a single pod (2026-09-24T12:52:44Z), self-recovering — the memory-stream receiver retries its polling cycle after the exception. No user-visible data loss is indicated; a transient delivery gap on one memory stream queue for at most one poll interval. It is filed because a 30 s grain stall is a symptom worth correlating if it recurs, not because the one event is an outage.
Where to look. src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs (the log site, GetQueueMessagesAsync), and the IMemoryStreamQueueGrain/MemoryStreamQueueGrain implementation for the memory-1 queue — investigate what the grain on 10.244.2.175 was executing around 2026-09-24T12:52Z. Correlate with Ops/Logs entries from that pod and the prior wedged-hub incidents above; the mesh docs Doc/Architecture/MeshNodeStreamCache and Doc/Architecture/LiveMirrorsAndTheChangeFeed describe the transport this queue carries. If this fingerprint recurs with growing occurrence counts, re-triage at a higher severity as a wedged-silo symptom.
Evidence
|
|
| Fingerprint |
a547847aca7e6357 |
| Category |
Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-1` |
| Severity |
Error |
| Exception |
System.TimeoutException |
| Namespace |
memex |
| Pods |
memex-portal-deployment-745745db7c-b2gjn |
| Occurrences |
1 |
| First seen |
2026-09-24 12:52:44Z |
| Last seen |
2026-09-24 12:52:44Z |
| Routing |
not determined — no configured route matches the category Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-1`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-24 12:52:44Z memex-portal-deployment-745745db7c-b2gjn fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-1[200402]
Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.80:11111:149257404 sys.svc.stream.agent/10.244.5.80:11111@149257404+Memory_1_memory-1-0x20000001]->[S10.244.2.175:11111:149249283 memorystreamqueue/be8eb06fb48875590100002001000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #36D24B4965682290.
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
Re-addressed by the current identity function: this incident inherited 643eee47bf4d00d2 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/a547847aca7e6357. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 6e99fc2ca7a39b38: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
What is failing. The Orleans memory-stream receiver for queue
memory-1on the memex portal pod (memex-portal-deployment-745745db7c-b2gjn) timed out waiting forIMemoryStreamQueueGrain.Dequeueon a peer silo (10.244.5.80 → 10.244.2.175): the grain did not answer within the 30 s response bound, soMemoryAdapterFactory.GetQueueMessagesAsyncthrew and the receiver's polling cycle for that queue faulted. This is the in-process streaming data plane (the per-silo cache/sync-stream transport behindIMeshNodeStreamCacheand the change feed), not application code.Probable cause — moderate confidence. The receiver itself is fine; the failure is on the far side: the
MemoryStreamQueueGrainhosting that queue was unresponsive for over 30 seconds. That is consistent with this fleet's known history of wedged silo-side actors under load — see the memex restart actionsOps/Actions/restart-memex-wedged-reads-20260921("wedged read hubs") andrestart-memex-confirmed-20260921(stale enumeration snapshot pumping a deleted partition) from three days before this burst. A grain stuck behind a long turn (e.g. a blocked hub round) would answer late or never within the response bound. The evidence does not positively identify which turn wedged the grain — cause at the grain level unclear; evidence points at silo-side congestion/wedged-actor contention on 10.244.2.175.Impact. Minimal as measured: a single occurrence on a single pod (2026-09-24T12:52:44Z), self-recovering — the memory-stream receiver retries its polling cycle after the exception. No user-visible data loss is indicated; a transient delivery gap on one memory stream queue for at most one poll interval. It is filed because a 30 s grain stall is a symptom worth correlating if it recurs, not because the one event is an outage.
Where to look.
src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs(the log site,GetQueueMessagesAsync), and theIMemoryStreamQueueGrain/MemoryStreamQueueGrainimplementation for thememory-1queue — investigate what the grain on 10.244.2.175 was executing around 2026-09-24T12:52Z. Correlate withOps/Logsentries from that pod and the prior wedged-hub incidents above; the mesh docsDoc/Architecture/MeshNodeStreamCacheandDoc/Architecture/LiveMirrorsAndTheChangeFeeddescribe the transport this queue carries. If this fingerprint recurs with growing occurrence counts, re-triage at a higher severity as a wedged-silo symptom.Evidence
a547847aca7e6357Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-1`System.TimeoutExceptionmemexmemex-portal-deployment-745745db7c-b2gjnOrleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-1`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.Recent log lines
Re-addressed by the current identity function: this incident inherited
643eee47bf4d00d2(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.Opened automatically from
Admin/_LogIncident/a547847aca7e6357. Recurrences are folded into this issue rather than opening new ones.It also stands for the whole log site
6e99fc2ca7a39b38: other fingerprints of this site fold in here as comments rather than opening tickets of their own.