Provision-plan log-entry write refused as ownership-undetermined when the NodeType definition-row read faults while the rest of the install succeeds

[systemorph-com]

What is failing

During phase 4/4 ("Record entries") of the memex-cloud provision plan, a single Hosting/LogEntry create under Ops/Logs was refused with the access.partitionCreate.undetermined failure: the read of the Hosting/LogEntry NodeType definition's durable row faulted or did not answer within the 10 s ProbeTimeout, so PartitionOwningTypes.OwnsPartitionWithoutActivating answered null and the create failed closed. The other 109 entries in the same window landed, and the window itself is preserved on the plan node (selector + counts), so the entry is recoverable by retry.

Probable cause — medium confidence.

This is the scenario Partition Ownership Resolution explicitly flags: IStorageAdapter.ReadMany and Exists/Write are independent interface members, and the design's "a store that cannot answer this read cannot answer the existence probe either" holds only per adapter stack — it was falsified once already by a test stand-in (Systemorph/MeshWeaver.Plugins#2047). Here the install's other writes in the same window succeeded (109/110 landed), so the fault/timeout was confined to the definition-row ReadMany — a transient partition-storage-hub fault or cold-hub latency on that one routed read, not a general store outage. Whether it is a one-off transient or a reproducible pairing break in the RoutingProxyAdapter.ReadMany path cannot be told from a single occurrence.

Impact

One occurrence on one pod (memex-portal-deployment-589ff8f895-vn49b); the provision plan is reported FAILED for the window but no data is lost — the selector and counts are on the plan node and the entry can be re-recorded. Secondary-path, retryable, easy workaround: re-run the phase.

Where to look

• PartitionOwningTypes.OwnsPartitionWithoutActivating / OwnsPartitionProvisioningValidator — the refusal site and the .Timeout(ProbeTimeout, null) / .Catch(_ => null) fold that hides fault vs timeout.
• RoutingProxyAdapter.ReadMany (and PersistenceService.ReadMany) — whether a transient partition-storage-hub fault can fault this read while Write/Exists on the same partition succeed, i.e. the Plugins#2047 lesson re-checked for the production adapter stack.
• The Hosting/LogEntry definition row read against the memex-cloud partition's storage hub around the first-seen timestamp.

Related — sibling incident Admin/_LogIncident/05f47831e3b7d6cd ("Install did not persist N node(s)") is the same ProvisionPlan log site and the burst this was folded from; that ticket tracks the general persist failures, this one the specific ownership-refusal shape. Likely the same underlying provisioning burst, but a distinct defect surface.

---

Evidence

Fingerprint	66c36a065c5e3baf
Category	ProvisionPlan
Severity	Error
Namespace	memex-cloud
Pods	memex-portal-deployment-589ff8f895-vn49b
Occurrences	1
First seen	2026-09-25 15:30:41Z
Last seen	2026-09-25 15:30:41Z
Routing	not determined — no configured route matches the category ProvisionPlan. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-25 15:30:41Z memex-portal-deployment-589ff8f895-vn49b fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex-cloud' FAILED in phase 4/4 'Record entries': the window was read (110 line(s)) and 109 of them landed in Ops/Logs; 1 could NOT be written — first fault: InvalidOperationException: Whether 'Hosting/LogEntry' owns its partition could not be established — the read of its NodeType definition failed, or did not answer within 10 s. 'Ops/Logs/memex-cloud-1790320472803631397-f8f895-ghgjt' was not created; please try again.. The selector and both counts are on this node, so the window is not lost

_{Re-addressed by the current identity function: this incident inherited 05f47831e3b7d6cd (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/66c36a065c5e3baf. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 5640ba5068fa6f6d: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

[rbuergi]

Triage (cluster with #5721). Probably the same teardown window, but not proven. Left open.

What I measured:

• The failure, 2026-09-25 15:30:41Z on …589ff8f895-vn49b, falls inside aks-ops run 36153998984, a Restart memex-cloud that started at 15:24:54Z and concluded success. vn49b belongs to the ReplicaSet being replaced: the Ops/logs-memexcloud-20260925-floor-vn49b run describes it as a ~30 h old ci.9291 process.
• On memex-cloud, the index shows Ops/logs-memexcloud-20260925-floor-vn49b last modified at 15:27:03Z (v12) and …-floor-ghgjt at 15:24:05Z (v13). Both are before the 15:30:41Z failure. The FAILED verdict of whichever run logged it never reached its node, which is the InstanceAction failure recording lost when the run hub is disposing — run stuck "in progress" #5721 shape: a hub being torn down cannot write.

What that suggests: the ownership probe's ReadMany of the Hosting/LogEntry definition row went unanswered for 10 s during that rollout. The partition storage hub or its pod was draining. OwnsPartitionWithoutActivating then correctly failed closed with "please try again". That refusal is the designed answer to a read that reached no verdict, and I do not read it as a defect in the probe.

What Systemorph/MeshWeaver.Plugins#2393 changes here: if the action's own hub was already IsShuttingDown when the phase failed, the plan's line is a Warning and the next activation records the run Unobserved.

Not established:

1. Whether this hub was shutting down at 15:30:41Z, or whether only the peer that owned the read was draining while this hub was live. In the second case the run is still Failed and logged at Error after The Orleans test host allocated a production-sized grain directory per silo (#2346) #2393.
2. Whether the probe's .Timeout(ProbeTimeout, null) / .Catch(_ => null) fold hid a ShuttingDown NACK. If it did, the refusal should say "the owner was recycling" instead of "could not be established", and an observation cut that way would fold to Unobserved like InstanceAction failure recording lost when the run hub is disposing — run stuck "in progress" #5721.

Answering (2) needs the portal log around 15:30:30–15:30:41Z on vn49b, via a Logs action on the control instance. I did not run one.

[systemorph-com]

Folded in — a DIFFERENT fingerprint from the same log site.

This issue tracks the log site 5640ba5068fa6f6d (ProvisionPlan), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.

Folded fingerprint	f088f1361a6b78c2
Log site	5640ba5068fa6f6d
Category	ProvisionPlan
Severity	Error
Exception	InvalidOperationException
Detail	ValidationFailed: Cannot delete '{value}': Validation error: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
Namespace	memex
Pods	memex-portal-deployment-5db78dfc99-d2cpk
Occurrences	1
First seen	2026-09-27 05:26:34Z
Last seen	2026-09-27 05:26:34Z
Already folded here	2 fingerprint(s)
Incident	Admin/_LogIncident/f088f1361a6b78c2

What triage wrote about this fingerprint

What is failing

The memex tenant's ProvisionPlan run (an 11-phase provisioning plan) failed in phase 9/11, Delete content. While recursively deleting the content root UWDeepfieldData (starting at UWDeepfieldData/Clients/SafeDriveInsurance), the delete validation step tried to resolve a service from an Autofac LifetimeScope that had already been disposed, so the whole delete — and the plan — aborted.

Probable cause (confidence: medium)

A disposed-scope resolution during a long-running recursive delete is the classic "hub was recycled mid-operation" pattern: the ProvisionPlan's delete phase holds onto a scope (or the hub's scope) across awaits, and the hub for UWDeepfieldData was quiesced/recycled while the delete was still walking the subtree; the next validation resolve then throws ObjectDisposedException-style Autofac error, wrapped as ValidationFailed: Cannot delete .... Supporting evidence: the same deployment family already logs MeshWeaver.Messaging.HubDisposingException / "the owning hub was disposed while the request was in flight" entries in Ops/Logs (e.g. the TestEmail delete thread from 2026-08-12, and many GetDataRequest ... owning hub was disposed entries from 2026-09-15), so hubs being torn down under live operations is a recurring shape in this codebase. Alternative (lower confidence): the delete phase creates a nested scope per subtree node and disposes it too early. Either way the defect is that the Delete content phase does not tolerate a hub/scope teardown mid-delete.

Impact

Low measured impact: 1 occurrence on 1 pod (memex-portal-deployment-5db78dfc99-d2cpk), first and last seen 2026-09-27T05:26:34Z. But the failure is operationally meaningful when it fires: the provisioning plan stops at phase 9, leaving the target content root partially deleted and the tenant in a half-provisioned state that requires a plan re-run. Note the target itself is real: UWDeepfieldData/Clients/SafeDriveInsurance — the uwdeepfielddata partition exists, so this was a live tenant teardown, not test noise.

Where to look

The Delete content phase handler of the ProvisionPlan component (log category ProvisionPlan, phase 9/11), and the delete-validation code path that resolves validators from the current LifetimeScope (the ValidationFailed: Cannot delete '{path}' wrapper). Compare with the idempotent-delete work already documented at Doc/Architecture/IdempotentDelete — that made deletes tolerate vanished nodes, but not disposed scopes. Likely fix directions: resolve validators from a fresh scope (or the root container) rather than the request scope, and/or catch the disposed-scope condition in the recursive delete and retry via the mesh address after the hub reactivates. Cause not confirmed from code — no ProvisionPlan source found in the mesh to pin the exact frame.

Still occurring: 1 more since the last update (1 total).

• Last seen: 2026-09-27 05:26:34Z
• Pods: memex-portal-deployment-5db78dfc99-d2cpk

Recent log lines

2026-09-27 05:26:34Z memex-portal-deployment-5db78dfc99-d2cpk fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 9/11 'Delete content': the delete of content root 'UWDeepfieldData' failed — InvalidOperationException: ValidationFailed: Cannot delete 'UWDeepfieldData/Clients/SafeDriveInsurance': Validation error: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.

[systemorph-com]

Folded in — a DIFFERENT fingerprint from the same log site.

This issue tracks the log site 5640ba5068fa6f6d (ProvisionPlan), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.

Folded fingerprint	f088f1361a6b78c2
Log site	5640ba5068fa6f6d
Category	ProvisionPlan
Severity	Error
Exception	InvalidOperationException
Detail	ValidationFailed: Cannot delete '{value}': Validation error: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
Namespace	memex
Pods	memex-portal-deployment-5db78dfc99-d2cpk
Occurrences	1
First seen	2026-09-27 05:26:34Z
Last seen	2026-09-27 05:26:34Z
Already folded here	2 fingerprint(s)
Incident	Admin/_LogIncident/f088f1361a6b78c2

What triage wrote about this fingerprint

What is failing

The memex tenant's ProvisionPlan run (an 11-phase provisioning plan) failed in phase 9/11, Delete content. While recursively deleting the content root UWDeepfieldData (starting at UWDeepfieldData/Clients/SafeDriveInsurance), the delete validation step tried to resolve a service from an Autofac LifetimeScope that had already been disposed, so the whole delete — and the plan — aborted.

Probable cause (confidence: medium)

A disposed-scope resolution during a long-running recursive delete is the classic "hub was recycled mid-operation" pattern: the ProvisionPlan's delete phase holds onto a scope (or the hub's scope) across awaits, and the hub for UWDeepfieldData was quiesced/recycled while the delete was still walking the subtree; the next validation resolve then throws ObjectDisposedException-style Autofac error, wrapped as ValidationFailed: Cannot delete .... Supporting evidence: the same deployment family already logs MeshWeaver.Messaging.HubDisposingException / "the owning hub was disposed while the request was in flight" entries in Ops/Logs (e.g. the TestEmail delete thread from 2026-08-12, and many GetDataRequest ... owning hub was disposed entries from 2026-09-15), so hubs being torn down under live operations is a recurring shape in this codebase. Alternative (lower confidence): the delete phase creates a nested scope per subtree node and disposes it too early. Either way the defect is that the Delete content phase does not tolerate a hub/scope teardown mid-delete.

Impact

Low measured impact: 1 occurrence on 1 pod (memex-portal-deployment-5db78dfc99-d2cpk), first and last seen 2026-09-27T05:26:34Z. But the failure is operationally meaningful when it fires: the provisioning plan stops at phase 9, leaving the target content root partially deleted and the tenant in a half-provisioned state that requires a plan re-run. Note the target itself is real: UWDeepfieldData/Clients/SafeDriveInsurance — the uwdeepfielddata partition exists, so this was a live tenant teardown, not test noise.

Where to look

The Delete content phase handler of the ProvisionPlan component (log category ProvisionPlan, phase 9/11), and the delete-validation code path that resolves validators from the current LifetimeScope (the ValidationFailed: Cannot delete '{path}' wrapper). Compare with the idempotent-delete work already documented at Doc/Architecture/IdempotentDelete — that made deletes tolerate vanished nodes, but not disposed scopes. Likely fix directions: resolve validators from a fresh scope (or the root container) rather than the request scope, and/or catch the disposed-scope condition in the recursive delete and retry via the mesh address after the hub reactivates. Cause not confirmed from code — no ProvisionPlan source found in the mesh to pin the exact frame.

Still occurring: 1 more since the last update (1 total).

• Last seen: 2026-09-27 05:26:34Z
• Pods: memex-portal-deployment-5db78dfc99-d2cpk

Recent log lines

2026-09-27 05:26:34Z memex-portal-deployment-5db78dfc99-d2cpk fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 9/11 'Delete content': the delete of content root 'UWDeepfieldData' failed — InvalidOperationException: ValidationFailed: Cannot delete 'UWDeepfieldData/Clients/SafeDriveInsurance': Validation error: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.

[systemorph-com]

Folded in — a DIFFERENT fingerprint from the same log site.

This issue tracks the log site 5640ba5068fa6f6d (ProvisionPlan), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.

Folded fingerprint	7956f1766cebea54
Log site	5640ba5068fa6f6d
Category	ProvisionPlan
Severity	Error
Exception	InvalidOperationException
Detail	Unavailable: Delete of '{value}' exceeded {n}s timeout in stage '{value}': [DeleteNode:pre-validate-descendants] the pre-flight fan-out for '{value}' did not settle within {n}s: {n} of {n} posted ValidateDeleteRequest(s) still outstanding ({n} descendant(s) planned, at most {n} in flight). Each leg carries its own {n}s bound and reports itself by name, so reaching THIS bound means the fan-out as a whole stalled, not that one leaf was slow: {path} (+{n} more)
Namespace	memex
Pods	memex-portal-deployment-5db78dfc99-tjd25
Occurrences	1
First seen	2026-09-27 06:17:20Z
Last seen	2026-09-27 06:17:20Z
Already folded here	3 fingerprint(s)
Incident	Admin/_LogIncident/7956f1766cebea54
Triage called it	Recursive delete's pre-validate stage bound is a duration cap that cannot fit large subtrees — a progressing fan-out is reported as stalled

What triage wrote about this fingerprint

What is failing

A provisioning plan's Delete content phase tried to delete the content root UWDeepfield (31,138 descendants) and was refused by the pre-validate-descendants stage: the delete never started (the pre-flight is atomic — nothing was removed), aborting the plan at phase 9/11.

Probable cause — medium-high confidence

The stage bound over the pre-flight fan-out is effectively a total-duration cap, not a no-progress watchdog — the same defect class already fixed for the commit stage. The numbers in the refusal show the fan-out was progressing normally when the cap fired: 1,740 of 31,138 legs posted, 64 in flight (the PreValidateFanOutConcurrency cap), each leg answering in roughly a second. At that rate the subtree needs ~7 minutes of healthy work, but the stage budget is 25s. Doc/Architecture/RecursiveDeleteDrain §4 documents the backstop's premise — "every leg terminates on its own, so reaching it means the fan-out as a whole stopped progressing" — and this occurrence falsifies that premise: with more planned legs than the budget admits at the concurrency cap, a perfectly healthy fan-out still reaches the bound. The fix that worked for commit (a tick per completed unit resetting the stage clock) was never applied to the pre-flight; the message text asserting "stalled" is also wrong in this regime and will misdirect whoever reads it.

Impact

One occurrence, one pod, one tenant deprovisioning. The target space is untouched (pre-flight refuses before any row is removed), so nothing is half-deleted; the plan phase aborts and the operator must retry. A workaround exists today — chunk the delete (as callers already do elsewhere) or retry — so this is a broken secondary/admin path on oversized subtrees, not a release blocker. It will recur deterministically for any space with tens of thousands of descendants.

Where to look

• src/MeshWeaver.Mesh.Contract/MeshExtensions.cs — HandleDeleteNodeRequest / PreValidateDescendantsObs, and how the stage bound wraps the capped fan-out; contrast with the tick-reset pattern already on the commit stage.
• The Recursive-Delete Drain §3–§4 — the commit-stage fix and the pre-flight's stated backstop contract.
• Related, not duplicates: A recursive delete's commit stage can stall for its whole budget — pre-flight fan-out and budget nesting are fixed and deployed; the commit stall is unmeasured on memex-cloud #1198 (commit-stage duration cap, same class), fix(delete): name the STAGE a delete timeout died in, and which descendant went silent #1294 (pre-flight report readability), Recursive delete is still aborted by a descendant that vanished between its plan and its pre-flight #4680 (absence confirmation) — none cover the stage bound scaling with subtree size.

Still occurring: 1 more since the last update (1 total).

• Last seen: 2026-09-27 06:17:20Z
• Pods: memex-portal-deployment-5db78dfc99-tjd25

Recent log lines

2026-09-27 06:17:20Z memex-portal-deployment-5db78dfc99-tjd25 fail: ProvisionPlan[0]
      [ProvisionPlan] 'memex' FAILED in phase 9/11 'Delete content': the delete of content root 'UWDeepfield' failed — InvalidOperationException: Unavailable: Delete of 'UWDeepfield' exceeded 25s timeout in stage 'pre-validate-descendants': [DeleteNode:pre-validate-descendants] the pre-flight fan-out for 'UWDeepfield' did not settle within 25s: 64 of 1740 posted ValidateDeleteRequest(s) still outstanding (31138 descendant(s) planned, at most 64 in flight). Each leg carries its own 20s bound and reports itself by name, so reaching THIS bound means the fan-out as a whole stalled, not that one leaf was slow: UWDeepfield/Broker/_Activity/compile-2026092305250231373d55ce11fe84ad1bfa4bd404800a145, UWDeepfield/CatastropheModel/Release/20260809190954-E8VCj3yA, UWDeepfield/CatastropheModel/_Activity/compile-202608131831083656ea0034a3b964a95aaaccea0371e2fec, UWDeepfield/Cedent/_Activity/compile-20260814100056217bb7332b8cb45490ab0bc22f59c8a02aa, UWDeepfield/ContractType/_Activity/compile-20260825011622969f31ce9bdb5244b53a19bdc33ad32ae80, UWDeepfield/CreditLimit/Release/20260812125659-VhrXCcy0, UWDeepfield/CreditLimit/_Activity/compile-2026080623555563696276681569247eb9812fc96b814d778, UWDeepfield/CrestaZone/_Activity/compile-20260807080830234bff9f77e4dc74aaab0710ffa3dff3134, UWDeepfield/Currency/Release/20260809040920-fBsTtWdx, UWDeepfield/Currency/Release/20260813162446-kXM9MWqC (+54 more)