Summary
On macOS (MoltenVK), the render loop intermittently loses the VkDevice with a GPU timeout, always surfaced through vkQueuePresentKHR, vkAcquireNextImageKHR, or occasionally vkQueueSubmit:
[mvk-error] VK_TIMEOUT: Lost VkDevice after MTLCommandBuffer "vkQueuePresentKHR MTLCommandBuffer on Queue 3-0" execution failed (code 2): Caused GPU Timeout Error (00000002:kIOGPUCommandBufferCallbackErrorTimeout)
[mvk-error] VK_TIMEOUT: Lost VkDevice after MTLCommandBuffer "vkAcquireNextImageKHR MTLCommandBuffer on Queue 3-0" execution failed (code 2): Caused GPU Timeout Error (00000002:kIOGPUCommandBufferCallbackErrorTimeout)
Once this fires, the whole VkDevice is dead — VulkanDevice::CheckDeviceLost/IsDeviceLost (added in this investigation) catches it and freezes the render loop safely instead of continuing to call into a dead device, but the underlying GPU timeout itself is still unexplained.
Never reproduces on Windows. Confirmed unrelated to the Windows-only timeline-semaphore corruption bug fixed in fix/dragdrop-material-texture-loading (RRM's mesh batch upload signalling DeviceSwapchain::RenderTimeline from two independent writers) — that fix is macOS-irrelevant (macOS never showed the semaphore-corruption symptom at all), and this GPU timeout kept recurring on macOS after that fix landed and was live-verified working correctly.
Reproduction
No reliable deterministic repro. Observed most often during/after extended interactive use of the editor (mesh drag-drop, multiple hot-reload swaps, UI interaction) — but also seen at near-idle. Engine log right before the crash consistently shows normal activity with no anomalies: correct, monotonically increasing timeline semaphore signal values, no validation errors, no asserts.
Investigated and ruled out this session
- Host-side synchronization theories: RRM's upload architecture (mesh batch uploads, texture uploads,
UpdateBuffer, UploadFontAtlas) was restructured multiple times during this investigation — device-loss detection, per-frame (not reused) command buffers, deferred staging-buffer frees via timeline gating instead of fence-then-immediate-free, a dedicated m_batch_timeline semaphore, and finally unifying all upload paths into one per-frame deferred batch. None of these changes affected the timeout's recurrence on macOS.
- MoltenVK version: confirmed up to date via checksum comparison against a freshly downloaded newer MoltenVK SDK build — the linked runtime matches.
- Vulkan-Headers/Loader vs. MoltenVK version skew: distinguished the Vulkan-Headers/Loader pin from MoltenVK's own runtime source; not a mismatch.
- A real, unrelated data race was found and fixed along the way:
VFSDiskBackend's file-object pool had no internal synchronization while VFSScanner fans its directory scan across ThreadPool workers — confirmed via a live PoolAllocator::Free crash once Metal API validation shifted timing enough to expose it. Fixed, but did not affect this timeout's recurrence.
- 5+ distinct architecture theories were raised and adversarially cleared (via independent verification passes) without landing on a root cause.
Complicating factor
The async spdlog sink consistently loses the last several log lines right before a crash of this kind, so the exact last Vulkan call/frame state immediately preceding the timeout can't be fully reconstructed from logs alone.
Current status / next step
Root cause is still unknown. Given repeated CPU-side host-synchronization audits haven't found it, the next productive step is likely a GPU frame capture (Xcode GPU capture or a MoltenVK/Metal-level trace) around a live repro, rather than further CPU-side log inspection — the same conclusion reached for the separate texture-rendering issue in #764.
Workaround in place
VulkanDevice::CheckDeviceLost + IsDeviceLost, wired into every QueueSubmit/Present/AcquireNextImage call site, plus a render-loop guard, so the engine freezes safely on this timeout instead of continuing to call into a dead device (which is UB and has been observed to segfault inside the Vulkan loader rather than return cleanly).
Summary
On macOS (MoltenVK), the render loop intermittently loses the VkDevice with a GPU timeout, always surfaced through
vkQueuePresentKHR,vkAcquireNextImageKHR, or occasionallyvkQueueSubmit:Once this fires, the whole VkDevice is dead —
VulkanDevice::CheckDeviceLost/IsDeviceLost(added in this investigation) catches it and freezes the render loop safely instead of continuing to call into a dead device, but the underlying GPU timeout itself is still unexplained.Never reproduces on Windows. Confirmed unrelated to the Windows-only timeline-semaphore corruption bug fixed in
fix/dragdrop-material-texture-loading(RRM's mesh batch upload signallingDeviceSwapchain::RenderTimelinefrom two independent writers) — that fix is macOS-irrelevant (macOS never showed the semaphore-corruption symptom at all), and this GPU timeout kept recurring on macOS after that fix landed and was live-verified working correctly.Reproduction
No reliable deterministic repro. Observed most often during/after extended interactive use of the editor (mesh drag-drop, multiple hot-reload swaps, UI interaction) — but also seen at near-idle. Engine log right before the crash consistently shows normal activity with no anomalies: correct, monotonically increasing timeline semaphore signal values, no validation errors, no asserts.
Investigated and ruled out this session
UpdateBuffer,UploadFontAtlas) was restructured multiple times during this investigation — device-loss detection, per-frame (not reused) command buffers, deferred staging-buffer frees via timeline gating instead of fence-then-immediate-free, a dedicatedm_batch_timelinesemaphore, and finally unifying all upload paths into one per-frame deferred batch. None of these changes affected the timeout's recurrence on macOS.VFSDiskBackend's file-object pool had no internal synchronization whileVFSScannerfans its directory scan across ThreadPool workers — confirmed via a livePoolAllocator::Freecrash once Metal API validation shifted timing enough to expose it. Fixed, but did not affect this timeout's recurrence.Complicating factor
The async spdlog sink consistently loses the last several log lines right before a crash of this kind, so the exact last Vulkan call/frame state immediately preceding the timeout can't be fully reconstructed from logs alone.
Current status / next step
Root cause is still unknown. Given repeated CPU-side host-synchronization audits haven't found it, the next productive step is likely a GPU frame capture (Xcode GPU capture or a MoltenVK/Metal-level trace) around a live repro, rather than further CPU-side log inspection — the same conclusion reached for the separate texture-rendering issue in #764.
Workaround in place
VulkanDevice::CheckDeviceLost+IsDeviceLost, wired into everyQueueSubmit/Present/AcquireNextImagecall site, plus a render-loop guard, so the engine freezes safely on this timeout instead of continuing to call into a dead device (which is UB and has been observed to segfault inside the Vulkan loader rather than return cleanly).