What happens with overprovisioning when there's no more space?

[refi64]

This is something I've been wondering about for a while. If my file systems are overprovisioned, but the pool runs out of physical space...what exactly happens? I've heard LVM's handling here is a bit messy, but I'm not sure what Stratis ends up doing.

There seems to be an existing open issue for this, but the now-implemented thin pool rework mentions changes to out-of-space handling too, so I'm not sure if the behavior is different now.

[mulkieran]

@refi64 Thanks for the question! We are at present working on a blog post about the thinpool rework that we have done; it should be up within a few days on our website. I hope that will answer your questions.

[refi64]

Would it be more useful to also send out a signal when the pool is about to run out of space to signal to stop writing or does that seem unlikely to be useful?

TBH that would be very useful! Off the top of my head, I would probably wire it up to just remount the rootfs as ro if possible, which certainly isn't particularly clean, but it's much easier to work with localized application data corruption than actual filesystem issues. (Off the top of my head, I'm admittedly also not sure what the prospect of that actually working is!)

[jbaublitz]

That is unfortunately going to be very hard to do as you can only unmount if you kill all of the daemons running on the system that are accessing the root filesystem. We had considered that as a safety feature, but it proved to not be feasible. This is largely why we put the no overprovisioning mode in place (for the root filesystem case). Regardless, we will be providing a new property soon that will send out a signal when it's changed per your request to notify you when you're at a critical level of free space!

[jbaublitz]

@refi64 I wanted to follow up on the signal indicating critical levels of free space.

First I wanted to outline what we currently do. Our general approach is to allocate blocks of data in 50 GiB chunks. Whenever the low water mark is reached, 50 GiB of data space will be allocated or if there is less than that available, as much as possible will be allocated to the data device. Our low water mark is 15 GiB of free space remaining. If there is no additional space to allocate and the low water mark is reached (15 GiB left), we set NoAllocSpace to true and send out a signal indicating that no more room is available to allocate to the data device.

My question for you is whether 15 GiB is a reasonable amount of space left for a critical space signal if there is no storage left to allocate. If so, then we already support this feature under the NoAllocSpace property change signal. If you would like a critical signal at a lower amount of free space, then we should discuss the properties of what you're expecting (as in how much space would you like to see left before we send the signal). Do you have any thoughts here?

[refi64]

15 GiB should work for me as-is then!

Interestingly enough, by complete coincidence, I did manage to exhaust disk space, and it created a giant mess of...nothing, actually. Attempting to write new files have ENOSPC, but I could still delete some old files to free up space, after which everything went back to normal. Does the worse behavior mostly happen if you have existing snapshots left, in which case deleting files wouldn't actually free up any space, or was I just really lucky?

[jbaublitz]

You would probably see the worst kinds of cases if the thin pool is full and an unclean unmount occurs. A lot of the scenarios that I've heard are very difficult to recover from are log replays with a full thin pool.